[PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
will introduce a new problem that when huge IP abnormal packet arrived,
it may cause OOM and break the kernel, just like this:

[   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
[  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
[  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
[  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
[  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
[  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
[  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
[  100.067050] Call Trace:
[  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
[  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
[  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
[  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
[  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
[  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
[  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
[  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
[  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
[  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
[  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
[  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
[  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
[  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
[  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
[  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
[  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
[  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
[  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
[  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
[  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
[  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
[  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140

================================cut here=====================================

The reason is that the huge abnormal IP packet will be received to net stack
and be dropped finally by dst_release, and the dst_release would use the rcuos
callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
calling do_softirq() to receive more and more IP abnormal packets which will be
throw into the RCU callbacks again later, the number of received packet is much
greater than the number of packets freed, it will exhaust the memory and then OOM,
so don't try to process any pending softirqs in the rcuos callback-offload kthread
is a more effective solution.

Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>

Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
---
 kernel/rcu/tree_plugin.h | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 85c5a88..760c3b5 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
 			if (__rcu_reclaim(rdp->rsp->name, list))
 				cl++;
 			c++;
-			local_bh_enable();
-			cond_resched_rcu_qs();
+			_local_bh_enable();
 			list = next;
 		}
 		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
-- 
1.9.0

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> will introduce a new problem that when huge IP abnormal packet arrived,
> it may cause OOM and break the kernel, just like this:
> 
> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
> [  100.067050] Call Trace:
> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> 
> ================================cut here=====================================
> 
> The reason is that the huge abnormal IP packet will be received to net stack
> and be dropped finally by dst_release, and the dst_release would use the rcuos
> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
> calling do_softirq() to receive more and more IP abnormal packets which will be
> throw into the RCU callbacks again later, the number of received packet is much
> greater than the number of packets freed, it will exhaust the memory and then OOM,
> so don't try to process any pending softirqs in the rcuos callback-offload kthread
> is a more effective solution.

OK, but we could still have softirqs processed by the grace-period kthread
as a result of any number of other events.  So this change might reduce
the probability of this problem, but it doesn't eliminate it.

How huge are these huge IP packets?  Is the underlying problem that they
are too large to use the memory-allocator fastpaths?

							Thanx, Paul

> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> 
> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> ---
>  kernel/rcu/tree_plugin.h | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 85c5a88..760c3b5 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>  			if (__rcu_reclaim(rdp->rsp->name, list))
>  				cl++;
>  			c++;
> -			local_bh_enable();
> -			cond_resched_rcu_qs();
> +			_local_bh_enable();
>  			list = next;
>  		}
>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
> -- 
> 1.9.0
> 
> 
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

On 2016/11/18 21:01, Paul E. McKenney wrote:
> On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
>> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>> will introduce a new problem that when huge IP abnormal packet arrived,
>> it may cause OOM and break the kernel, just like this:
>>
>> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
>> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
>> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
>> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
>> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
>> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
>> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
>> [  100.067050] Call Trace:
>> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
>> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
>> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
>> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
>> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
>> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
>> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
>> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
>> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
>> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
>> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
>> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
>> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
>> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
>> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
>> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
>> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
>> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
>> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
>> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
>> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
>> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>
>> ================================cut here=====================================
>>
>> The reason is that the huge abnormal IP packet will be received to net stack
>> and be dropped finally by dst_release, and the dst_release would use the rcuos
>> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
>> calling do_softirq() to receive more and more IP abnormal packets which will be
>> throw into the RCU callbacks again later, the number of received packet is much
>> greater than the number of packets freed, it will exhaust the memory and then OOM,
>> so don't try to process any pending softirqs in the rcuos callback-offload kthread
>> is a more effective solution.
> 
> OK, but we could still have softirqs processed by the grace-period kthread
> as a result of any number of other events.  So this change might reduce
> the probability of this problem, but it doesn't eliminate it.
> 
> How huge are these huge IP packets?  Is the underlying problem that they
> are too large to use the memory-allocator fastpaths?
> 
> 							Thanx, Paul
> 

I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
IP abnormal packet to full speed.

The Mac abnormal packet would be dropped at low level and not be received to net stack,
but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
release later by dst_release because it is meaningless.

dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);

so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.

Thanks.
Ding


>> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>
>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>> ---
>>  kernel/rcu/tree_plugin.h | 3 +--
>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>> index 85c5a88..760c3b5 100644
>> --- a/kernel/rcu/tree_plugin.h
>> +++ b/kernel/rcu/tree_plugin.h
>> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>  			if (__rcu_reclaim(rdp->rsp->name, list))
>>  				cl++;
>>  			c++;
>> -			local_bh_enable();
>> -			cond_resched_rcu_qs();
>> +			_local_bh_enable();
>>  			list = next;
>>  		}
>>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
>> -- 
>> 1.9.0
>>
>>
>>
> 
> 
> .
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
> 
> 
> On 2016/11/18 21:01, Paul E. McKenney wrote:
> > On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
> >> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> >> will introduce a new problem that when huge IP abnormal packet arrived,
> >> it may cause OOM and break the kernel, just like this:
> >>
> >> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
> >> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
> >> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
> >> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
> >> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
> >> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
> >> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
> >> [  100.067050] Call Trace:
> >> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
> >> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
> >> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
> >> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
> >> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
> >> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
> >> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
> >> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
> >> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
> >> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
> >> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
> >> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
> >> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
> >> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
> >> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
> >> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
> >> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
> >> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
> >> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
> >> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
> >> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> >> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
> >> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> >>
> >> ================================cut here=====================================
> >>
> >> The reason is that the huge abnormal IP packet will be received to net stack
> >> and be dropped finally by dst_release, and the dst_release would use the rcuos
> >> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
> >> calling do_softirq() to receive more and more IP abnormal packets which will be
> >> throw into the RCU callbacks again later, the number of received packet is much
> >> greater than the number of packets freed, it will exhaust the memory and then OOM,
> >> so don't try to process any pending softirqs in the rcuos callback-offload kthread
> >> is a more effective solution.
> > 
> > OK, but we could still have softirqs processed by the grace-period kthread
> > as a result of any number of other events.  So this change might reduce
> > the probability of this problem, but it doesn't eliminate it.
> > 
> > How huge are these huge IP packets?  Is the underlying problem that they
> > are too large to use the memory-allocator fastpaths?
> > 
> > 							Thanx, Paul
> > 
> 
> I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
> IP abnormal packet to full speed.
> 
> The Mac abnormal packet would be dropped at low level and not be received to net stack,
> but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
> release later by dst_release because it is meaningless.
> 
> dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
> 
> so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
> if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
> so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
> fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.

The overall effect of these two patches is to move from enabling bh
(and processing recent softirqs) to enabling bh without processing
recent softirqs.  Is this really the correct way to solve this problem?
What about this solution is avoiding re-introducing the original
softlockups?  Have you talked to the networking guys about this issue?

							Thanx, Paul

> Thanks.
> Ding
> 
> 
> >> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> >> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> >>
> >> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> >> ---
> >>  kernel/rcu/tree_plugin.h | 3 +--
> >>  1 file changed, 1 insertion(+), 2 deletions(-)
> >>
> >> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> >> index 85c5a88..760c3b5 100644
> >> --- a/kernel/rcu/tree_plugin.h
> >> +++ b/kernel/rcu/tree_plugin.h
> >> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> >>  			if (__rcu_reclaim(rdp->rsp->name, list))
> >>  				cl++;
> >>  			c++;
> >> -			local_bh_enable();
> >> -			cond_resched_rcu_qs();
> >> +			_local_bh_enable();
> >>  			list = next;
> >>  		}
> >>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
> >> -- 
> >> 1.9.0
> >>
> >>
> >>
> > 
> > 
> > .
> > 
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Sat, Nov 19, 2016 at 12:22:09AM -0800, Paul E. McKenney wrote:
> On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
> > 
> > 
> > On 2016/11/18 21:01, Paul E. McKenney wrote:
> > > On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
> > >> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> > >> will introduce a new problem that when huge IP abnormal packet arrived,
> > >> it may cause OOM and break the kernel, just like this:
> > >>
> > >> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
> > >> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
> > >> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
> > >> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
> > >> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
> > >> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
> > >> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
> > >> [  100.067050] Call Trace:
> > >> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
> > >> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
> > >> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
> > >> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
> > >> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
> > >> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
> > >> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
> > >> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
> > >> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
> > >> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
> > >> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
> > >> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
> > >> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
> > >> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
> > >> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
> > >> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
> > >> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
> > >> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
> > >> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
> > >> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
> > >> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> > >> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
> > >> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> > >>
> > >> ================================cut here=====================================
> > >>
> > >> The reason is that the huge abnormal IP packet will be received to net stack
> > >> and be dropped finally by dst_release, and the dst_release would use the rcuos
> > >> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
> > >> calling do_softirq() to receive more and more IP abnormal packets which will be
> > >> throw into the RCU callbacks again later, the number of received packet is much
> > >> greater than the number of packets freed, it will exhaust the memory and then OOM,
> > >> so don't try to process any pending softirqs in the rcuos callback-offload kthread
> > >> is a more effective solution.
> > > 
> > > OK, but we could still have softirqs processed by the grace-period kthread
> > > as a result of any number of other events.  So this change might reduce
> > > the probability of this problem, but it doesn't eliminate it.
> > > 
> > > How huge are these huge IP packets?  Is the underlying problem that they
> > > are too large to use the memory-allocator fastpaths?
> > > 
> > > 							Thanx, Paul
> > > 
> > 
> > I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
> > IP abnormal packet to full speed.
> > 
> > The Mac abnormal packet would be dropped at low level and not be received to net stack,
> > but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
> > release later by dst_release because it is meaningless.
> > 
> > dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
> > 
> > so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
> > if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
> > so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
> > fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.

And following up on my fastpath point -- from what I can see, one
big effect of the large invalid packets is that they push processing
off of a number of fastpaths.  If these packets could be rejected with
less per-packet processing, I bet that things would work much better.

						Thanx, Paul

> The overall effect of these two patches is to move from enabling bh
> (and processing recent softirqs) to enabling bh without processing
> recent softirqs.  Is this really the correct way to solve this problem?
> What about this solution is avoiding re-introducing the original
> softlockups?  Have you talked to the networking guys about this issue?
> 
> 							Thanx, Paul
> 
> > Thanks.
> > Ding
> > 
> > 
> > >> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> > >> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> > >>
> > >> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> > >> ---
> > >>  kernel/rcu/tree_plugin.h | 3 +--
> > >>  1 file changed, 1 insertion(+), 2 deletions(-)
> > >>
> > >> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> > >> index 85c5a88..760c3b5 100644
> > >> --- a/kernel/rcu/tree_plugin.h
> > >> +++ b/kernel/rcu/tree_plugin.h
> > >> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> > >>  			if (__rcu_reclaim(rdp->rsp->name, list))
> > >>  				cl++;
> > >>  			c++;
> > >> -			local_bh_enable();
> > >> -			cond_resched_rcu_qs();
> > >> +			_local_bh_enable();
> > >>  			list = next;
> > >>  		}
> > >>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
> > >> -- 
> > >> 1.9.0
> > >>
> > >>
> > >>
> > > 
> > > 
> > > .
> > > 
> > 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

On 2016/11/21 8:13, Paul E. McKenney wrote:
> On Sat, Nov 19, 2016 at 12:22:09AM -0800, Paul E. McKenney wrote:
>> On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
>>>
>>>
>>> On 2016/11/18 21:01, Paul E. McKenney wrote:
>>>> On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
>>>>> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>>>>> will introduce a new problem that when huge IP abnormal packet arrived,
>>>>> it may cause OOM and break the kernel, just like this:
>>>>>
>>>>> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
>>>>> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
>>>>> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
>>>>> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
>>>>> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
>>>>> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
>>>>> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
>>>>> [  100.067050] Call Trace:
>>>>> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
>>>>> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
>>>>> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
>>>>> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
>>>>> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
>>>>> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
>>>>> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
>>>>> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
>>>>> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
>>>>> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
>>>>> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
>>>>> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
>>>>> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
>>>>> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
>>>>> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
>>>>> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
>>>>> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
>>>>> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
>>>>> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
>>>>> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
>>>>> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>>>> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
>>>>> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>>>>
>>>>> ================================cut here=====================================
>>>>>
>>>>> The reason is that the huge abnormal IP packet will be received to net stack
>>>>> and be dropped finally by dst_release, and the dst_release would use the rcuos
>>>>> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
>>>>> calling do_softirq() to receive more and more IP abnormal packets which will be
>>>>> throw into the RCU callbacks again later, the number of received packet is much
>>>>> greater than the number of packets freed, it will exhaust the memory and then OOM,
>>>>> so don't try to process any pending softirqs in the rcuos callback-offload kthread
>>>>> is a more effective solution.
>>>>
>>>> OK, but we could still have softirqs processed by the grace-period kthread
>>>> as a result of any number of other events.  So this change might reduce
>>>> the probability of this problem, but it doesn't eliminate it.
>>>>
>>>> How huge are these huge IP packets?  Is the underlying problem that they
>>>> are too large to use the memory-allocator fastpaths?
>>>>
>>>> 							Thanx, Paul
>>>>
>>>
>>> I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
>>> IP abnormal packet to full speed.
>>>
>>> The Mac abnormal packet would be dropped at low level and not be received to net stack,
>>> but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
>>> release later by dst_release because it is meaningless.
>>>
>>> dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
>>>
>>> so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
>>> if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
>>> so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
>>> fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.
> 
> And following up on my fastpath point -- from what I can see, one
> big effect of the large invalid packets is that they push processing
> off of a number of fastpaths.  If these packets could be rejected with
> less per-packet processing, I bet that things would work much better.
> 
> 						Thanx, Paul

Yes, and I found the WARN_ON_ONCE(!irqs_disabled()) will be triggered if use _local_bh_enable here,
so I think we could ask some help from Eric and David how to reject the huge number packets.

Thanks
Ding

> 
>> The overall effect of these two patches is to move from enabling bh
>> (and processing recent softirqs) to enabling bh without processing
>> recent softirqs.  Is this really the correct way to solve this problem?
>> What about this solution is avoiding re-introducing the original
>> softlockups?  Have you talked to the networking guys about this issue?
>>
>> 							Thanx, Paul
>>
>>> Thanks.
>>> Ding
>>>
>>>
>>>>> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>>>>
>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>>>> ---
>>>>>  kernel/rcu/tree_plugin.h | 3 +--
>>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>>>>> index 85c5a88..760c3b5 100644
>>>>> --- a/kernel/rcu/tree_plugin.h
>>>>> +++ b/kernel/rcu/tree_plugin.h
>>>>> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>>>>  			if (__rcu_reclaim(rdp->rsp->name, list))
>>>>>  				cl++;
>>>>>  			c++;
>>>>> -			local_bh_enable();
>>>>> -			cond_resched_rcu_qs();
>>>>> +			_local_bh_enable();
>>>>>  			list = next;
>>>>>  		}
>>>>>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
>>>>> -- 
>>>>> 1.9.0
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
> 
> 
> .
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

Hi, Paul:

I try to debug this problem and found this solution could work well for both problem scene.


diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 85c5a88..dbc14a7 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
                        if (__rcu_reclaim(rdp->rsp->name, list))
                                cl++;
                        c++;
-                   local_bh_enable();
+                 _local_bh_enable();
                        cond_resched_rcu_qs();
                        list = next;
                }


The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
what do you think about it? Please give me some suggestion.

Thanks.
Ding

On 2016/11/21 9:28, Ding Tianhong wrote:
> 
> 
> On 2016/11/21 8:13, Paul E. McKenney wrote:
>> On Sat, Nov 19, 2016 at 12:22:09AM -0800, Paul E. McKenney wrote:
>>> On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
>>>>
>>>>
>>>> On 2016/11/18 21:01, Paul E. McKenney wrote:
>>>>> On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
>>>>>> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>>>>>> will introduce a new problem that when huge IP abnormal packet arrived,
>>>>>> it may cause OOM and break the kernel, just like this:
>>>>>>
>>>>>> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
>>>>>> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
>>>>>> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
>>>>>> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
>>>>>> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
>>>>>> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
>>>>>> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
>>>>>> [  100.067050] Call Trace:
>>>>>> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
>>>>>> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
>>>>>> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
>>>>>> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
>>>>>> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
>>>>>> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
>>>>>> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
>>>>>> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
>>>>>> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
>>>>>> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
>>>>>> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
>>>>>> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
>>>>>> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
>>>>>> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
>>>>>> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
>>>>>> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
>>>>>> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
>>>>>> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
>>>>>> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
>>>>>> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
>>>>>> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>>>>> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
>>>>>> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>>>>>
>>>>>> ================================cut here=====================================
>>>>>>
>>>>>> The reason is that the huge abnormal IP packet will be received to net stack
>>>>>> and be dropped finally by dst_release, and the dst_release would use the rcuos
>>>>>> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
>>>>>> calling do_softirq() to receive more and more IP abnormal packets which will be
>>>>>> throw into the RCU callbacks again later, the number of received packet is much
>>>>>> greater than the number of packets freed, it will exhaust the memory and then OOM,
>>>>>> so don't try to process any pending softirqs in the rcuos callback-offload kthread
>>>>>> is a more effective solution.
>>>>>
>>>>> OK, but we could still have softirqs processed by the grace-period kthread
>>>>> as a result of any number of other events.  So this change might reduce
>>>>> the probability of this problem, but it doesn't eliminate it.
>>>>>
>>>>> How huge are these huge IP packets?  Is the underlying problem that they
>>>>> are too large to use the memory-allocator fastpaths?
>>>>>
>>>>> 							Thanx, Paul
>>>>>
>>>>
>>>> I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
>>>> IP abnormal packet to full speed.
>>>>
>>>> The Mac abnormal packet would be dropped at low level and not be received to net stack,
>>>> but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
>>>> release later by dst_release because it is meaningless.
>>>>
>>>> dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
>>>>
>>>> so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
>>>> if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
>>>> so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
>>>> fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.
>>
>> And following up on my fastpath point -- from what I can see, one
>> big effect of the large invalid packets is that they push processing
>> off of a number of fastpaths.  If these packets could be rejected with
>> less per-packet processing, I bet that things would work much better.
>>
>> 						Thanx, Paul
> 
> Yes, and I found the WARN_ON_ONCE(!irqs_disabled()) will be triggered if use _local_bh_enable here,
> so I think we could ask some help from Eric and David how to reject the huge number packets.
> 
> Thanks
> Ding
> 
>>
>>> The overall effect of these two patches is to move from enabling bh
>>> (and processing recent softirqs) to enabling bh without processing
>>> recent softirqs.  Is this really the correct way to solve this problem?
>>> What about this solution is avoiding re-introducing the original
>>> softlockups?  Have you talked to the networking guys about this issue?
>>>
>>> 							Thanx, Paul
>>>
>>>> Thanks.
>>>> Ding
>>>>
>>>>
>>>>>> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>>>>>
>>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>>>>> ---
>>>>>>  kernel/rcu/tree_plugin.h | 3 +--
>>>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>>>>>> index 85c5a88..760c3b5 100644
>>>>>> --- a/kernel/rcu/tree_plugin.h
>>>>>> +++ b/kernel/rcu/tree_plugin.h
>>>>>> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>>>>>  			if (__rcu_reclaim(rdp->rsp->name, list))
>>>>>>  				cl++;
>>>>>>  			c++;
>>>>>> -			local_bh_enable();
>>>>>> -			cond_resched_rcu_qs();
>>>>>> +			_local_bh_enable();
>>>>>>  			list = next;
>>>>>>  		}
>>>>>>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
>>>>>> -- 
>>>>>> 1.9.0
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>
>>
>> .
>>

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
> Hi, Paul:
> 
> I try to debug this problem and found this solution could work well for both problem scene.
> 
> 
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 85c5a88..dbc14a7 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>                         if (__rcu_reclaim(rdp->rsp->name, list))
>                                 cl++;
>                         c++;
> -                   local_bh_enable();
> +                 _local_bh_enable();
>                         cond_resched_rcu_qs();
>                         list = next;
>                 }
> 
> 
> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
> what do you think about it? Please give me some suggestion.

>From what I can see, there is absolutely no guarantee that
cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
that it will process any pending softirqs -- and that is not part of
its job in any case.  So I cannot recommend the above patch.

On efficient handling of large invalid packets (that is still the issue,
right?), I must defer to Dave and Eric.

								Thanx, Paul

> Thanks.
> Ding
> 
> On 2016/11/21 9:28, Ding Tianhong wrote:
> > 
> > 
> > On 2016/11/21 8:13, Paul E. McKenney wrote:
> >> On Sat, Nov 19, 2016 at 12:22:09AM -0800, Paul E. McKenney wrote:
> >>> On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
> >>>>
> >>>>
> >>>> On 2016/11/18 21:01, Paul E. McKenney wrote:
> >>>>> On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
> >>>>>> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> >>>>>> will introduce a new problem that when huge IP abnormal packet arrived,
> >>>>>> it may cause OOM and break the kernel, just like this:
> >>>>>>
> >>>>>> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
> >>>>>> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
> >>>>>> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
> >>>>>> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
> >>>>>> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
> >>>>>> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
> >>>>>> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
> >>>>>> [  100.067050] Call Trace:
> >>>>>> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
> >>>>>> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
> >>>>>> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
> >>>>>> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
> >>>>>> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
> >>>>>> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
> >>>>>> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
> >>>>>> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
> >>>>>> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
> >>>>>> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
> >>>>>> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
> >>>>>> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
> >>>>>> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
> >>>>>> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
> >>>>>> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
> >>>>>> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
> >>>>>> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
> >>>>>> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
> >>>>>> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
> >>>>>> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
> >>>>>> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> >>>>>> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
> >>>>>> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> >>>>>>
> >>>>>> ================================cut here=====================================
> >>>>>>
> >>>>>> The reason is that the huge abnormal IP packet will be received to net stack
> >>>>>> and be dropped finally by dst_release, and the dst_release would use the rcuos
> >>>>>> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
> >>>>>> calling do_softirq() to receive more and more IP abnormal packets which will be
> >>>>>> throw into the RCU callbacks again later, the number of received packet is much
> >>>>>> greater than the number of packets freed, it will exhaust the memory and then OOM,
> >>>>>> so don't try to process any pending softirqs in the rcuos callback-offload kthread
> >>>>>> is a more effective solution.
> >>>>>
> >>>>> OK, but we could still have softirqs processed by the grace-period kthread
> >>>>> as a result of any number of other events.  So this change might reduce
> >>>>> the probability of this problem, but it doesn't eliminate it.
> >>>>>
> >>>>> How huge are these huge IP packets?  Is the underlying problem that they
> >>>>> are too large to use the memory-allocator fastpaths?
> >>>>>
> >>>>> 							Thanx, Paul
> >>>>>
> >>>>
> >>>> I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
> >>>> IP abnormal packet to full speed.
> >>>>
> >>>> The Mac abnormal packet would be dropped at low level and not be received to net stack,
> >>>> but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
> >>>> release later by dst_release because it is meaningless.
> >>>>
> >>>> dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
> >>>>
> >>>> so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
> >>>> if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
> >>>> so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
> >>>> fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.
> >>
> >> And following up on my fastpath point -- from what I can see, one
> >> big effect of the large invalid packets is that they push processing
> >> off of a number of fastpaths.  If these packets could be rejected with
> >> less per-packet processing, I bet that things would work much better.
> >>
> >> 						Thanx, Paul
> > 
> > Yes, and I found the WARN_ON_ONCE(!irqs_disabled()) will be triggered if use _local_bh_enable here,
> > so I think we could ask some help from Eric and David how to reject the huge number packets.
> > 
> > Thanks
> > Ding
> > 
> >>
> >>> The overall effect of these two patches is to move from enabling bh
> >>> (and processing recent softirqs) to enabling bh without processing
> >>> recent softirqs.  Is this really the correct way to solve this problem?
> >>> What about this solution is avoiding re-introducing the original
> >>> softlockups?  Have you talked to the networking guys about this issue?
> >>>
> >>> 							Thanx, Paul
> >>>
> >>>> Thanks.
> >>>> Ding
> >>>>
> >>>>
> >>>>>> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> >>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> >>>>>>
> >>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> >>>>>> ---
> >>>>>>  kernel/rcu/tree_plugin.h | 3 +--
> >>>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
> >>>>>>
> >>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> >>>>>> index 85c5a88..760c3b5 100644
> >>>>>> --- a/kernel/rcu/tree_plugin.h
> >>>>>> +++ b/kernel/rcu/tree_plugin.h
> >>>>>> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> >>>>>>  			if (__rcu_reclaim(rdp->rsp->name, list))
> >>>>>>  				cl++;
> >>>>>>  			c++;
> >>>>>> -			local_bh_enable();
> >>>>>> -			cond_resched_rcu_qs();
> >>>>>> +			_local_bh_enable();
> >>>>>>  			list = next;
> >>>>>>  		}
> >>>>>>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
> >>>>>> -- 
> >>>>>> 1.9.0
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >>
> >>
> >> .
> >>
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote:
> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
> > Hi, Paul:
> > 
> > I try to debug this problem and found this solution could work well for both problem scene.
> > 
> > 
> > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> > index 85c5a88..dbc14a7 100644
> > --- a/kernel/rcu/tree_plugin.h
> > +++ b/kernel/rcu/tree_plugin.h
> > @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> >                         if (__rcu_reclaim(rdp->rsp->name, list))
> >                                 cl++;
> >                         c++;
> > -                   local_bh_enable();
> > +                 _local_bh_enable();
> >                         cond_resched_rcu_qs();
> >                         list = next;
> >                 }
> > 
> > 
> > The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
> > local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
> > what do you think about it? Please give me some suggestion.
> 
> From what I can see, there is absolutely no guarantee that
> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
> that it will process any pending softirqs -- and that is not part of
> its job in any case.  So I cannot recommend the above patch.
> 
> On efficient handling of large invalid packets (that is still the issue,
> right?), I must defer to Dave and Eric.

On the perhaps unlikely off-chance that there is a fix for this outside
of networking, what symptoms are you seeing without this fix in place?
Still RCU CPU stall warnings?  Soft lockups?  Something else?

								Thanx, Paul

> > Thanks.
> > Ding
> > 
> > On 2016/11/21 9:28, Ding Tianhong wrote:
> > > 
> > > 
> > > On 2016/11/21 8:13, Paul E. McKenney wrote:
> > >> On Sat, Nov 19, 2016 at 12:22:09AM -0800, Paul E. McKenney wrote:
> > >>> On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
> > >>>>
> > >>>>
> > >>>> On 2016/11/18 21:01, Paul E. McKenney wrote:
> > >>>>> On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
> > >>>>>> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> > >>>>>> will introduce a new problem that when huge IP abnormal packet arrived,
> > >>>>>> it may cause OOM and break the kernel, just like this:
> > >>>>>>
> > >>>>>> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
> > >>>>>> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
> > >>>>>> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
> > >>>>>> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
> > >>>>>> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
> > >>>>>> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
> > >>>>>> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
> > >>>>>> [  100.067050] Call Trace:
> > >>>>>> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
> > >>>>>> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
> > >>>>>> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
> > >>>>>> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
> > >>>>>> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
> > >>>>>> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
> > >>>>>> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
> > >>>>>> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
> > >>>>>> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
> > >>>>>> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
> > >>>>>> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
> > >>>>>> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
> > >>>>>> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
> > >>>>>> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
> > >>>>>> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
> > >>>>>> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
> > >>>>>> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
> > >>>>>> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
> > >>>>>> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
> > >>>>>> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
> > >>>>>> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> > >>>>>> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
> > >>>>>> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
> > >>>>>>
> > >>>>>> ================================cut here=====================================
> > >>>>>>
> > >>>>>> The reason is that the huge abnormal IP packet will be received to net stack
> > >>>>>> and be dropped finally by dst_release, and the dst_release would use the rcuos
> > >>>>>> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
> > >>>>>> calling do_softirq() to receive more and more IP abnormal packets which will be
> > >>>>>> throw into the RCU callbacks again later, the number of received packet is much
> > >>>>>> greater than the number of packets freed, it will exhaust the memory and then OOM,
> > >>>>>> so don't try to process any pending softirqs in the rcuos callback-offload kthread
> > >>>>>> is a more effective solution.
> > >>>>>
> > >>>>> OK, but we could still have softirqs processed by the grace-period kthread
> > >>>>> as a result of any number of other events.  So this change might reduce
> > >>>>> the probability of this problem, but it doesn't eliminate it.
> > >>>>>
> > >>>>> How huge are these huge IP packets?  Is the underlying problem that they
> > >>>>> are too large to use the memory-allocator fastpaths?
> > >>>>>
> > >>>>> 							Thanx, Paul
> > >>>>>
> > >>>>
> > >>>> I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
> > >>>> IP abnormal packet to full speed.
> > >>>>
> > >>>> The Mac abnormal packet would be dropped at low level and not be received to net stack,
> > >>>> but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
> > >>>> release later by dst_release because it is meaningless.
> > >>>>
> > >>>> dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
> > >>>>
> > >>>> so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
> > >>>> if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
> > >>>> so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
> > >>>> fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.
> > >>
> > >> And following up on my fastpath point -- from what I can see, one
> > >> big effect of the large invalid packets is that they push processing
> > >> off of a number of fastpaths.  If these packets could be rejected with
> > >> less per-packet processing, I bet that things would work much better.
> > >>
> > >> 						Thanx, Paul
> > > 
> > > Yes, and I found the WARN_ON_ONCE(!irqs_disabled()) will be triggered if use _local_bh_enable here,
> > > so I think we could ask some help from Eric and David how to reject the huge number packets.
> > > 
> > > Thanks
> > > Ding
> > > 
> > >>
> > >>> The overall effect of these two patches is to move from enabling bh
> > >>> (and processing recent softirqs) to enabling bh without processing
> > >>> recent softirqs.  Is this really the correct way to solve this problem?
> > >>> What about this solution is avoiding re-introducing the original
> > >>> softlockups?  Have you talked to the networking guys about this issue?
> > >>>
> > >>> 							Thanx, Paul
> > >>>
> > >>>> Thanks.
> > >>>> Ding
> > >>>>
> > >>>>
> > >>>>>> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
> > >>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> > >>>>>>
> > >>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
> > >>>>>> ---
> > >>>>>>  kernel/rcu/tree_plugin.h | 3 +--
> > >>>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
> > >>>>>>
> > >>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> > >>>>>> index 85c5a88..760c3b5 100644
> > >>>>>> --- a/kernel/rcu/tree_plugin.h
> > >>>>>> +++ b/kernel/rcu/tree_plugin.h
> > >>>>>> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> > >>>>>>  			if (__rcu_reclaim(rdp->rsp->name, list))
> > >>>>>>  				cl++;
> > >>>>>>  			c++;
> > >>>>>> -			local_bh_enable();
> > >>>>>> -			cond_resched_rcu_qs();
> > >>>>>> +			_local_bh_enable();
> > >>>>>>  			list = next;
> > >>>>>>  		}
> > >>>>>>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
> > >>>>>> -- 
> > >>>>>> 1.9.0
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> .
> > >>>>>
> > >>>>
> > >>
> > >>
> > >> .
> > >>
> > 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

On 2017/1/4 8:57, Paul E. McKenney wrote:
> On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote:
>> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
>>> Hi, Paul:
>>>
>>> I try to debug this problem and found this solution could work well for both problem scene.
>>>
>>>
>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>>> index 85c5a88..dbc14a7 100644
>>> --- a/kernel/rcu/tree_plugin.h
>>> +++ b/kernel/rcu/tree_plugin.h
>>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>>                         if (__rcu_reclaim(rdp->rsp->name, list))
>>>                                 cl++;
>>>                         c++;
>>> -                   local_bh_enable();
>>> +                 _local_bh_enable();
>>>                         cond_resched_rcu_qs();
>>>                         list = next;
>>>                 }
>>>
>>>
>>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
>>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
>>> what do you think about it? Please give me some suggestion.
>>
>> From what I can see, there is absolutely no guarantee that
>> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
>> that it will process any pending softirqs -- and that is not part of
>> its job in any case.  So I cannot recommend the above patch.
>>
>> On efficient handling of large invalid packets (that is still the issue,
>> right?), I must defer to Dave and Eric.
> 
> On the perhaps unlikely off-chance that there is a fix for this outside
> of networking, what symptoms are you seeing without this fix in place?
> Still RCU CPU stall warnings?  Soft lockups?  Something else?
> 
> 								Thanx, Paul
> 

Hi Paul:

I was still try to test and fix this by another way, but could explain more about this problem.

when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu),
so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will
call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run,
it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to
be freed.
So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough.

and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet.

Thanks
Ding



>>> Thanks.
>>> Ding
>>>
>>> On 2016/11/21 9:28, Ding Tianhong wrote:
>>>>
>>>>
>>>> On 2016/11/21 8:13, Paul E. McKenney wrote:
>>>>> On Sat, Nov 19, 2016 at 12:22:09AM -0800, Paul E. McKenney wrote:
>>>>>> On Sat, Nov 19, 2016 at 03:50:32PM +0800, Ding Tianhong wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2016/11/18 21:01, Paul E. McKenney wrote:
>>>>>>>> On Fri, Nov 18, 2016 at 08:40:09PM +0800, Ding Tianhong wrote:
>>>>>>>>> The commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>>>>>>>>> will introduce a new problem that when huge IP abnormal packet arrived,
>>>>>>>>> it may cause OOM and break the kernel, just like this:
>>>>>>>>>
>>>>>>>>> [   79.441538] mlx4_en: eth5: Leaving promiscuous mode steering mode:2
>>>>>>>>> [  100.067032] ksoftirqd/0: page allocation failure: order:0, mode:0x120
>>>>>>>>> [  100.067038] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ----V-------   3.10.0-327.28.3.28.x86_64 #1
>>>>>>>>> [  100.067039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20161018_184732-HGH1000003483 04/01/2014
>>>>>>>>> [  100.067041]  0000000000000120 00000000b080d798 ffff8802afd5b968 ffffffff81638cb9
>>>>>>>>> [  100.067045]  ffff8802afd5b9f8 ffffffff81171380 0000000000000010 0000000000000000
>>>>>>>>> [  100.067048]  ffff8802befd8000 00000000ffffffff 0000000000000001 00000000b080d798
>>>>>>>>> [  100.067050] Call Trace:
>>>>>>>>> [  100.067057]  [<ffffffff81638cb9>] dump_stack+0x19/0x1b
>>>>>>>>> [  100.067062]  [<ffffffff81171380>] warn_alloc_failed+0x110/0x180
>>>>>>>>> [  100.067066]  [<ffffffff81175b16>] __alloc_pages_nodemask+0x9b6/0xba0
>>>>>>>>> [  100.067070]  [<ffffffff8151e400>] ? skb_add_rx_frag+0x90/0xb0
>>>>>>>>> [  100.067075]  [<ffffffff811b6fba>] alloc_pages_current+0xaa/0x170
>>>>>>>>> [  100.067080]  [<ffffffffa06b9be0>] mlx4_alloc_pages.isra.24+0x40/0x170 [mlx4_en]
>>>>>>>>> [  100.067083]  [<ffffffffa06b9dec>] mlx4_en_alloc_frags+0xdc/0x220 [mlx4_en]
>>>>>>>>> [  100.067086]  [<ffffffff8152eeb8>] ? __netif_receive_skb+0x18/0x60
>>>>>>>>> [  100.067088]  [<ffffffff8152ef40>] ? netif_receive_skb+0x40/0xc0
>>>>>>>>> [  100.067092]  [<ffffffffa06bb521>] mlx4_en_process_rx_cq+0x5f1/0xec0 [mlx4_en]
>>>>>>>>> [  100.067095]  [<ffffffff8131027d>] ? list_del+0xd/0x30
>>>>>>>>> [  100.067098]  [<ffffffff8152c90f>] ? __napi_complete+0x1f/0x30
>>>>>>>>> [  100.067101]  [<ffffffffa06bbeef>] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
>>>>>>>>> [  100.067103]  [<ffffffff8152f372>] net_rx_action+0x152/0x240
>>>>>>>>> [  100.067107]  [<ffffffff81084d1f>] __do_softirq+0xef/0x280
>>>>>>>>> [  100.067109]  [<ffffffff81084ee0>] run_ksoftirqd+0x30/0x50
>>>>>>>>> [  100.067114]  [<ffffffff810ae93f>] smpboot_thread_fn+0xff/0x1a0
>>>>>>>>> [  100.067117]  [<ffffffff8163e269>] ? schedule+0x29/0x70
>>>>>>>>> [  100.067120]  [<ffffffff810ae840>] ? lg_double_unlock+0x90/0x90
>>>>>>>>> [  100.067122]  [<ffffffff810a5d4f>] kthread+0xcf/0xe0
>>>>>>>>> [  100.067124]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>>>>>>>> [  100.067127]  [<ffffffff81649198>] ret_from_fork+0x58/0x90
>>>>>>>>> [  100.067129]  [<ffffffff810a5c80>] ? kthread_create_on_node+0x140/0x140
>>>>>>>>>
>>>>>>>>> ================================cut here=====================================
>>>>>>>>>
>>>>>>>>> The reason is that the huge abnormal IP packet will be received to net stack
>>>>>>>>> and be dropped finally by dst_release, and the dst_release would use the rcuos
>>>>>>>>> callback-offload kthread to free the packet, but the cond_resched_rcu_qs() will
>>>>>>>>> calling do_softirq() to receive more and more IP abnormal packets which will be
>>>>>>>>> throw into the RCU callbacks again later, the number of received packet is much
>>>>>>>>> greater than the number of packets freed, it will exhaust the memory and then OOM,
>>>>>>>>> so don't try to process any pending softirqs in the rcuos callback-offload kthread
>>>>>>>>> is a more effective solution.
>>>>>>>>
>>>>>>>> OK, but we could still have softirqs processed by the grace-period kthread
>>>>>>>> as a result of any number of other events.  So this change might reduce
>>>>>>>> the probability of this problem, but it doesn't eliminate it.
>>>>>>>>
>>>>>>>> How huge are these huge IP packets?  Is the underlying problem that they
>>>>>>>> are too large to use the memory-allocator fastpaths?
>>>>>>>>
>>>>>>>> 							Thanx, Paul
>>>>>>>>
>>>>>>>
>>>>>>> I use the 40G mellanox NiC to receive packet, and the testgine could send Mac abnormal packet and
>>>>>>> IP abnormal packet to full speed.
>>>>>>>
>>>>>>> The Mac abnormal packet would be dropped at low level and not be received to net stack,
>>>>>>> but the IP abnormal packet will introduce this problem, every packet will looks as new dst first and
>>>>>>> release later by dst_release because it is meaningless.
>>>>>>>
>>>>>>> dst_release->call_rcu(&dst->rcu_head, dst_destroy_rcu);
>>>>>>>
>>>>>>> so all packet will be freed until the rcuos callback-offload kthread processing, it will be a infinite loop
>>>>>>> if huge packet is coming because the do_softirq will load more and more packet to the rcuos processing kthread,
>>>>>>> so I still could not find a better way to fix this, btw, it is really hard to say the driver use too large memory-allocater
>>>>>>> fastpaths, there is no memory leak and the Ixgbe may meet the same problem too.
>>>>>
>>>>> And following up on my fastpath point -- from what I can see, one
>>>>> big effect of the large invalid packets is that they push processing
>>>>> off of a number of fastpaths.  If these packets could be rejected with
>>>>> less per-packet processing, I bet that things would work much better.
>>>>>
>>>>> 						Thanx, Paul
>>>>
>>>> Yes, and I found the WARN_ON_ONCE(!irqs_disabled()) will be triggered if use _local_bh_enable here,
>>>> so I think we could ask some help from Eric and David how to reject the huge number packets.
>>>>
>>>> Thanks
>>>> Ding
>>>>
>>>>>
>>>>>> The overall effect of these two patches is to move from enabling bh
>>>>>> (and processing recent softirqs) to enabling bh without processing
>>>>>> recent softirqs.  Is this really the correct way to solve this problem?
>>>>>> What about this solution is avoiding re-introducing the original
>>>>>> softlockups?  Have you talked to the networking guys about this issue?
>>>>>>
>>>>>> 							Thanx, Paul
>>>>>>
>>>>>>> Thanks.
>>>>>>> Ding
>>>>>>>
>>>>>>>
>>>>>>>>> Fix commit bedc196915 ("rcu: Fix soft lockup for rcu_nocb_kthread")
>>>>>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>>>>>>>>
>>>>>>>>> Signed-off-by: Ding Tianhong <dingtianhong@huawei.com>
>>>>>>>>> ---
>>>>>>>>>  kernel/rcu/tree_plugin.h | 3 +--
>>>>>>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>>>>>>>>> index 85c5a88..760c3b5 100644
>>>>>>>>> --- a/kernel/rcu/tree_plugin.h
>>>>>>>>> +++ b/kernel/rcu/tree_plugin.h
>>>>>>>>> @@ -2172,8 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>>>>>>>>  			if (__rcu_reclaim(rdp->rsp->name, list))
>>>>>>>>>  				cl++;
>>>>>>>>>  			c++;
>>>>>>>>> -			local_bh_enable();
>>>>>>>>> -			cond_resched_rcu_qs();
>>>>>>>>> +			_local_bh_enable();
>>>>>>>>>  			list = next;
>>>>>>>>>  		}
>>>>>>>>>  		trace_rcu_batch_end(rdp->rsp->name, c, !!list, 0, 0, 1);
>>>>>>>>> -- 
>>>>>>>>> 1.9.0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> .
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>
> 
> 
> .
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote:
> 
> 
> On 2017/1/4 8:57, Paul E. McKenney wrote:
> > On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote:
> >> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
> >>> Hi, Paul:
> >>>
> >>> I try to debug this problem and found this solution could work well for both problem scene.
> >>>
> >>>
> >>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> >>> index 85c5a88..dbc14a7 100644
> >>> --- a/kernel/rcu/tree_plugin.h
> >>> +++ b/kernel/rcu/tree_plugin.h
> >>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> >>>                         if (__rcu_reclaim(rdp->rsp->name, list))
> >>>                                 cl++;
> >>>                         c++;
> >>> -                   local_bh_enable();
> >>> +                 _local_bh_enable();
> >>>                         cond_resched_rcu_qs();
> >>>                         list = next;
> >>>                 }
> >>>
> >>>
> >>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
> >>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
> >>> what do you think about it? Please give me some suggestion.
> >>
> >> From what I can see, there is absolutely no guarantee that
> >> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
> >> that it will process any pending softirqs -- and that is not part of
> >> its job in any case.  So I cannot recommend the above patch.
> >>
> >> On efficient handling of large invalid packets (that is still the issue,
> >> right?), I must defer to Dave and Eric.
> > 
> > On the perhaps unlikely off-chance that there is a fix for this outside
> > of networking, what symptoms are you seeing without this fix in place?
> > Still RCU CPU stall warnings?  Soft lockups?  Something else?
> > 
> > 								Thanx, Paul
> > 
> 
> Hi Paul:
> 
> I was still try to test and fix this by another way, but could explain more about this problem.
> 
> when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu),
> so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will
> call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run,
> it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to
> be freed.
> So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough.
> 
> and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet.

OK, I don't see any reasonable way that the RCU callback-offload tasks
(rcuos) can figure out whether or not they should let softirqs happen --
unconditionally suppressing them might help your workload, but would
break workloads needing low networking latency, of which there are many.

So please let me know now things go with Eric's patch.

							Thanx, Paul

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

On 2017/1/4 21:48, Paul E. McKenney wrote:
> On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote:
>>
>>
>> On 2017/1/4 8:57, Paul E. McKenney wrote:
>>> On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote:
>>>> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
>>>>> Hi, Paul:
>>>>>
>>>>> I try to debug this problem and found this solution could work well for both problem scene.
>>>>>
>>>>>
>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>>>>> index 85c5a88..dbc14a7 100644
>>>>> --- a/kernel/rcu/tree_plugin.h
>>>>> +++ b/kernel/rcu/tree_plugin.h
>>>>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>>>>                         if (__rcu_reclaim(rdp->rsp->name, list))
>>>>>                                 cl++;
>>>>>                         c++;
>>>>> -                   local_bh_enable();
>>>>> +                 _local_bh_enable();
>>>>>                         cond_resched_rcu_qs();
>>>>>                         list = next;
>>>>>                 }
>>>>>
>>>>>
>>>>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
>>>>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
>>>>> what do you think about it? Please give me some suggestion.
>>>>
>>>> From what I can see, there is absolutely no guarantee that
>>>> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
>>>> that it will process any pending softirqs -- and that is not part of
>>>> its job in any case.  So I cannot recommend the above patch.
>>>>
>>>> On efficient handling of large invalid packets (that is still the issue,
>>>> right?), I must defer to Dave and Eric.
>>>
>>> On the perhaps unlikely off-chance that there is a fix for this outside
>>> of networking, what symptoms are you seeing without this fix in place?
>>> Still RCU CPU stall warnings?  Soft lockups?  Something else?
>>>
>>> 								Thanx, Paul
>>>
>>
>> Hi Paul:
>>
>> I was still try to test and fix this by another way, but could explain more about this problem.
>>
>> when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu),
>> so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will
>> call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run,
>> it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to
>> be freed.
>> So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough.
>>
>> and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet.
> 
> OK, I don't see any reasonable way that the RCU callback-offload tasks
> (rcuos) can figure out whether or not they should let softirqs happen --
> unconditionally suppressing them might help your workload, but would
> break workloads needing low networking latency, of which there are many.
> 
> So please let me know now things go with Eric's patch.
> 
Hi Paul:

Good news, the Eric's patch could fix this problem, it means that if the softirqd kthread is running, we should not take too much
time in the softirq process, this behavior equivalent that we remove the do_softirq in the local_bh_enable(), but this solution looks more
perfect, we need to inform the lts kernel maintainer to applied this patch which is not looks like a bugfix.

Thanks
Ding


> 							Thanx, Paul
> 
> 
> .
> 

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Paul E. McKenney]

On Tue, Jan 10, 2017 at 11:20:40AM +0800, Ding Tianhong wrote:
> 
> 
> On 2017/1/4 21:48, Paul E. McKenney wrote:
> > On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote:
> >>
> >>
> >> On 2017/1/4 8:57, Paul E. McKenney wrote:
> >>> On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote:
> >>>> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
> >>>>> Hi, Paul:
> >>>>>
> >>>>> I try to debug this problem and found this solution could work well for both problem scene.
> >>>>>
> >>>>>
> >>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> >>>>> index 85c5a88..dbc14a7 100644
> >>>>> --- a/kernel/rcu/tree_plugin.h
> >>>>> +++ b/kernel/rcu/tree_plugin.h
> >>>>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
> >>>>>                         if (__rcu_reclaim(rdp->rsp->name, list))
> >>>>>                                 cl++;
> >>>>>                         c++;
> >>>>> -                   local_bh_enable();
> >>>>> +                 _local_bh_enable();
> >>>>>                         cond_resched_rcu_qs();
> >>>>>                         list = next;
> >>>>>                 }
> >>>>>
> >>>>>
> >>>>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
> >>>>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
> >>>>> what do you think about it? Please give me some suggestion.
> >>>>
> >>>> From what I can see, there is absolutely no guarantee that
> >>>> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
> >>>> that it will process any pending softirqs -- and that is not part of
> >>>> its job in any case.  So I cannot recommend the above patch.
> >>>>
> >>>> On efficient handling of large invalid packets (that is still the issue,
> >>>> right?), I must defer to Dave and Eric.
> >>>
> >>> On the perhaps unlikely off-chance that there is a fix for this outside
> >>> of networking, what symptoms are you seeing without this fix in place?
> >>> Still RCU CPU stall warnings?  Soft lockups?  Something else?
> >>>
> >>> 								Thanx, Paul
> >>>
> >>
> >> Hi Paul:
> >>
> >> I was still try to test and fix this by another way, but could explain more about this problem.
> >>
> >> when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu),
> >> so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will
> >> call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run,
> >> it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to
> >> be freed.
> >> So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough.
> >>
> >> and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet.
> > 
> > OK, I don't see any reasonable way that the RCU callback-offload tasks
> > (rcuos) can figure out whether or not they should let softirqs happen --
> > unconditionally suppressing them might help your workload, but would
> > break workloads needing low networking latency, of which there are many.
> > 
> > So please let me know now things go with Eric's patch.
> > 
> Hi Paul:
> 
> Good news, the Eric's patch could fix this problem, it means that if the softirqd kthread is running, we should not take too much
> time in the softirq process, this behavior equivalent that we remove the do_softirq in the local_bh_enable(), but this solution looks more
> perfect, we need to inform the lts kernel maintainer to applied this patch which is not looks like a bugfix.

Here is hoping!  ;-)

							Thanx, Paul

Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic

[Ding Tianhong]

Hi David：

The Patch "rcu: Fix soft lockup for rcu_nocb_kthread" has been added to several stable tree， it may introduced an issue in certain special scenarios,
The Patch "softirq: Let ksoftirqd do its job" could fix this issue, so I hope you could add this patch to stable tree, thanks.

Ding

On 2017/1/10 13:51, Paul E. McKenney wrote:
> On Tue, Jan 10, 2017 at 11:20:40AM +0800, Ding Tianhong wrote:
>>
>>
>> On 2017/1/4 21:48, Paul E. McKenney wrote:
>>> On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote:
>>>>
>>>>
>>>> On 2017/1/4 8:57, Paul E. McKenney wrote:
>>>>> On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote:
>>>>>> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote:
>>>>>>> Hi, Paul:
>>>>>>>
>>>>>>> I try to debug this problem and found this solution could work well for both problem scene.
>>>>>>>
>>>>>>>
>>>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
>>>>>>> index 85c5a88..dbc14a7 100644
>>>>>>> --- a/kernel/rcu/tree_plugin.h
>>>>>>> +++ b/kernel/rcu/tree_plugin.h
>>>>>>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg)
>>>>>>>                         if (__rcu_reclaim(rdp->rsp->name, list))
>>>>>>>                                 cl++;
>>>>>>>                         c++;
>>>>>>> -                   local_bh_enable();
>>>>>>> +                 _local_bh_enable();
>>>>>>>                         cond_resched_rcu_qs();
>>>>>>>                         list = next;
>>>>>>>                 }
>>>>>>>
>>>>>>>
>>>>>>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use
>>>>>>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives,
>>>>>>> what do you think about it? Please give me some suggestion.
>>>>>>
>>>>>> From what I can see, there is absolutely no guarantee that
>>>>>> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee
>>>>>> that it will process any pending softirqs -- and that is not part of
>>>>>> its job in any case.  So I cannot recommend the above patch.
>>>>>>
>>>>>> On efficient handling of large invalid packets (that is still the issue,
>>>>>> right?), I must defer to Dave and Eric.
>>>>>
>>>>> On the perhaps unlikely off-chance that there is a fix for this outside
>>>>> of networking, what symptoms are you seeing without this fix in place?
>>>>> Still RCU CPU stall warnings?  Soft lockups?  Something else?
>>>>>
>>>>> 								Thanx, Paul
>>>>>
>>>>
>>>> Hi Paul:
>>>>
>>>> I was still try to test and fix this by another way, but could explain more about this problem.
>>>>
>>>> when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu),
>>>> so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will
>>>> call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run,
>>>> it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to
>>>> be freed.
>>>> So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough.
>>>>
>>>> and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet.
>>>
>>> OK, I don't see any reasonable way that the RCU callback-offload tasks
>>> (rcuos) can figure out whether or not they should let softirqs happen --
>>> unconditionally suppressing them might help your workload, but would
>>> break workloads needing low networking latency, of which there are many.
>>>
>>> So please let me know now things go with Eric's patch.
>>>
>> Hi Paul:
>>
>> Good news, the Eric's patch could fix this problem, it means that if the softirqd kthread is running, we should not take too much
>> time in the softirq process, this behavior equivalent that we remove the do_softirq in the local_bh_enable(), but this solution looks more
>> perfect, we need to inform the lts kernel maintainer to applied this patch which is not looks like a bugfix.
> 
> Here is hoping!  ;-)
> 
> 							Thanx, Paul
> 
> 
> .
> 

[lkp] [rcu] 83ee00c6cf: WARNING:at_kernel/softirq.c:#__local_bh_enable

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 3182 bytes --]


FYI, we noticed the following commit:

https://github.com/0day-ci/linux Ding-Tianhong/rcu-fix-the-OOM-problem-of-huge-IP-abnormal-packet-traffic/20161118-204521
commit 83ee00c6cf5eaa85f74094d6800732edf7114ef9 ("rcu: fix the OOM problem of huge IP abnormal packet traffic")

in testcase: boot

on test machine: qemu-system-x86_64 -enable-kvm -m 320M

caused below changes:


+------------------------------------------------+------------+------------+
|                                                | 68ad1194cf | 83ee00c6cf |
+------------------------------------------------+------------+------------+
| boot_successes                                 | 6          | 0          |
| boot_failures                                  | 0          | 6          |
| WARNING:at_kernel/softirq.c:#__local_bh_enable | 0          | 6          |
| calltrace:_local_bh_enable                     | 0          | 6          |
+------------------------------------------------+------------+------------+



[    0.846125] PCI: CLS 0 bytes, default 64
[    0.847479] Unpacking initramfs...
[    0.849690] ------------[ cut here ]------------
[    0.850615] WARNING: CPU: 0 PID: 9 at kernel/softirq.c:140 __local_bh_enable+0x35/0x41
[    0.852518] Modules linked in:
[    0.853178] CPU: 0 PID: 9 Comm: rcuos/0 Not tainted 4.9.0-rc1-00041-g83ee00c #1
[    0.854630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[    0.856628]  ffff8800002f7c70 ffffffff81267760 0000000000000000 ffffffff81837340
[    0.858185]  ffff8800002f7cb0 ffffffff81060e07 0000008c00000009 0000000000000200
[    0.859742]  ffff8800002f7dc8 ffff88000f808bb0 0000000000000001 ffff88000f808bb0
[    0.861293] Call Trace:
[    0.861795]  [<ffffffff81267760>] dump_stack+0x61/0x7e
[    0.862809]  [<ffffffff81060e07>] __warn+0xf5/0x110
[    0.863870]  [<ffffffff81060f85>] warn_slowpath_null+0x18/0x1a
[    0.865020]  [<ffffffff81066b09>] __local_bh_enable+0x35/0x41
[    0.866143]  [<ffffffff81066b52>] _local_bh_enable+0x3d/0x3f
[    0.867252]  [<ffffffff810dbf14>] rcu_nocb_kthread+0x69b/0x6f2
[    0.868393]  [<ffffffff811cbe81>] ? __d_free_external+0x3f/0x3f
[    0.869554]  [<ffffffff810db879>] ? note_gp_changes+0xcd/0xcd
[    0.870679]  [<ffffffff815ff413>] ? __schedule+0x5fc/0x73c
[    0.871755]  [<ffffffff810db879>] ? note_gp_changes+0xcd/0xcd
[    0.872980]  [<ffffffff810898c7>] kthread+0x191/0x1a0
[    0.873971]  [<ffffffff81089736>] ? kthread_park+0x5d/0x5d
[    0.875059]  [<ffffffff8108fd54>] ? finish_task_switch+0x1e4/0x2a0
[    0.876262]  [<ffffffff81089736>] ? kthread_park+0x5d/0x5d
[    0.877331]  [<ffffffff81089736>] ? kthread_park+0x5d/0x5d
[    0.878401]  [<ffffffff81606a45>] ret_from_fork+0x25/0x30
[    0.879484] ---[ end trace 825c5dbf85ebfadd ]---
[    0.899723] workqueue: round-robin CPU selection forced, expect performance impact
[    2.115863] Freeing initrd memory: 9088K (ffff880013700000 - ffff880013fe0000)


To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp qemu -k <bzImage> job-script  # job-script is attached in this email



Thanks,
Xiaolong

[-- Attachment #2: config-4.9.0-rc1-00041-g83ee00c --]
[-- Type: text/plain, Size: 83381 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 4.9.0-rc1 Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=28
CONFIG_ARCH_MMAP_RND_BITS_MAX=32
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_KASAN_SHADOW_OFFSET=0xdffffc0000000000
CONFIG_X86_64_SMP=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_DEBUG_RODATA=y
CONFIG_PGTABLE_LEVELS=4
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
CONFIG_KERNEL_LZO=y
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_FHANDLE=y
CONFIG_USELIB=y
# CONFIG_AUDIT is not set
CONFIG_HAVE_ARCH_AUDITSYSCALL=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
# CONFIG_IRQ_DOMAIN_DEBUG is not set
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
CONFIG_RCU_EXPERT=y
CONFIG_SRCU=y
CONFIG_TASKS_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_KTHREAD_PRIO=0
CONFIG_RCU_NOCB_CPU=y
# CONFIG_RCU_NOCB_CPU_NONE is not set
# CONFIG_RCU_NOCB_CPU_ZERO is not set
CONFIG_RCU_NOCB_CPU_ALL=y
CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_NMI_LOG_BUF_SHIFT=13
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_ARCH_SUPPORTS_INT128=y
# CONFIG_NUMA_BALANCING is not set
CONFIG_CGROUPS=y
# CONFIG_MEMCG is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
# CONFIG_CGROUP_PIDS is not set
# CONFIG_CGROUP_FREEZER is not set
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=y
# CONFIG_CGROUP_CPUACCT is not set
# CONFIG_CGROUP_PERF is not set
# CONFIG_CGROUP_DEBUG is not set
# CONFIG_CHECKPOINT_RESTORE is not set
# CONFIG_NAMESPACES is not set
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
# CONFIG_RD_LZ4 is not set
# CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BPF=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_MULTIUSER=y
# CONFIG_SGETMASK_SYSCALL is not set
CONFIG_SYSFS_SYSCALL=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PRINTK=y
CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
# CONFIG_ELF_CORE is not set
# CONFIG_PCSPKR_PLATFORM is not set
# CONFIG_BASE_FULL is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
# CONFIG_BPF_SYSCALL is not set
CONFIG_SHMEM=y
# CONFIG_AIO is not set
CONFIG_ADVISE_SYSCALLS=y
# CONFIG_USERFAULTFD is not set
CONFIG_PCI_QUIRKS=y
CONFIG_MEMBARRIER=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_USE_VMALLOC=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_DEBUG_PERF_USE_VMALLOC=y
CONFIG_VM_EVENT_COUNTERS=y
# CONFIG_SLUB_DEBUG is not set
CONFIG_COMPAT_BRK=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_SLAB_FREELIST_RANDOM=y
# CONFIG_SLUB_CPU_PARTIAL is not set
# CONFIG_SYSTEM_DATA_VERIFICATION is not set
CONFIG_PROFILING=y
CONFIG_KEXEC_CORE=y
# CONFIG_OPROFILE is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
# CONFIG_KPROBES is not set
# CONFIG_JUMP_LABEL is not set
# CONFIG_UPROBES is not set
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_NMI=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_CLK=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_GCC_PLUGINS=y
# CONFIG_GCC_PLUGINS is not set
CONFIG_HAVE_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR is not set
CONFIG_CC_STACKPROTECTOR_NONE=y
# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_HAVE_EXIT_THREAD=y
CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_HAVE_STACK_VALIDATION=y
# CONFIG_HAVE_ARCH_HASH is not set
CONFIG_ISA_BUS_API=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y
# CONFIG_CPU_NO_EFFICIENT_FFS is not set
CONFIG_HAVE_ARCH_VMAP_STACK=y

#
# GCOV-based kernel profiling
#
CONFIG_GCOV_KERNEL=y
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# CONFIG_GCOV_PROFILE_ALL is not set
# CONFIG_GCOV_FORMAT_AUTODETECT is not set
# CONFIG_GCOV_FORMAT_3_4 is not set
CONFIG_GCOV_FORMAT_4_7=y
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=1
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
CONFIG_MODULE_COMPRESS=y
# CONFIG_MODULE_COMPRESS_GZIP is not set
CONFIG_MODULE_COMPRESS_XZ=y
CONFIG_MODULES_TREE_LOOKUP=y
# CONFIG_BLOCK is not set
CONFIG_PADATA=y
CONFIG_ASN1=m
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_QUEUED_RWLOCKS=y
CONFIG_FREEZER=y

#
# Processor type and features
#
# CONFIG_ZONE_DMA is not set
CONFIG_SMP=y
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_FAST_FEATURE_TESTS=y
# CONFIG_X86_X2APIC is not set
CONFIG_X86_MPPARSE=y
CONFIG_GOLDFISH=y
# CONFIG_X86_EXTENDED_PLATFORM is not set
# CONFIG_X86_INTEL_LPSS is not set
# CONFIG_X86_AMD_PLATFORM_DEVICE is not set
CONFIG_IOSF_MBI=y
# CONFIG_IOSF_MBI_DEBUG is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_DEBUG is not set
# CONFIG_PARAVIRT_SPINLOCKS is not set
# CONFIG_XEN is not set
CONFIG_KVM_GUEST=y
# CONFIG_KVM_DEBUG_FS is not set
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
CONFIG_PARAVIRT_CLOCK=y
CONFIG_NO_BOOTMEM=y
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PROCESSOR_SELECT=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
# CONFIG_CPU_SUP_CENTAUR is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=64
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not set
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set

#
# Performance monitoring
#
# CONFIG_PERF_EVENTS_INTEL_UNCORE is not set
CONFIG_PERF_EVENTS_INTEL_RAPL=m
# CONFIG_PERF_EVENTS_INTEL_CSTATE is not set
CONFIG_PERF_EVENTS_AMD_POWER=m
# CONFIG_VM86 is not set
CONFIG_X86_VSYSCALL_EMULATION=y
CONFIG_I8K=m
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
# CONFIG_X86_CPUID is not set
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_X86_DIRECT_GBPAGES=y
CONFIG_NUMA=y
# CONFIG_AMD_NUMA is not set
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=6
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
CONFIG_MOVABLE_NODE=y
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_MEMORY_BALLOON=y
# CONFIG_BALLOON_COMPACTION is not set
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
# CONFIG_HWPOISON_INJECT is not set
# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_CLEANCACHE=y
CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
CONFIG_CMA_DEBUGFS=y
CONFIG_CMA_AREAS=7
CONFIG_ZPOOL=m
CONFIG_ZBUD=y
CONFIG_Z3FOLD=m
# CONFIG_ZSMALLOC is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
CONFIG_ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT=y
# CONFIG_IDLE_PAGE_TRACKING is not set
CONFIG_FRAME_VECTOR=y
CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
CONFIG_ARCH_HAS_PKEYS=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set
CONFIG_X86_RESERVE_LOW=64
# CONFIG_MTRR is not set
# CONFIG_ARCH_RANDOM is not set
CONFIG_X86_SMAP=y
# CONFIG_X86_INTEL_MPX is not set
CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
# CONFIG_RANDOMIZE_BASE is not set
CONFIG_PHYSICAL_ALIGN=0x200000
CONFIG_HOTPLUG_CPU=y
# CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
CONFIG_DEBUG_HOTPLUG_CPU0=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_LEGACY_VSYSCALL_NATIVE is not set
CONFIG_LEGACY_VSYSCALL_EMULATE=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set
# CONFIG_MODIFY_LDT_SYSCALL is not set
CONFIG_HAVE_LIVEPATCH=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_SUSPEND_SKIP_SYNC is not set
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_AUTOSLEEP=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
# CONFIG_PM_WAKELOCKS_GC is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_DPM_WATCHDOG is not set
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_PM_CLK=y
CONFIG_PM_GENERIC_DOMAINS=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
CONFIG_PM_GENERIC_DOMAINS_SLEEP=y
CONFIG_PM_GENERIC_DOMAINS_OF=y
CONFIG_ACPI=y
CONFIG_ACPI_LEGACY_TABLES_LOOKUP=y
CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y
CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y
# CONFIG_ACPI_DEBUGGER is not set
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_REV_OVERRIDE_POSSIBLE=y
# CONFIG_ACPI_EC_DEBUGFS is not set
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_CPU_FREQ_PSS=y
CONFIG_ACPI_PROCESSOR_CSTATE=y
CONFIG_ACPI_PROCESSOR_IDLE=y
CONFIG_ACPI_PROCESSOR=y
# CONFIG_ACPI_IPMI is not set
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_TABLE_UPGRADE=y
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_IOAPIC=y
# CONFIG_ACPI_SBS is not set
# CONFIG_ACPI_HED is not set
# CONFIG_ACPI_CUSTOM_METHOD is not set
# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
CONFIG_HAVE_ACPI_APEI=y
CONFIG_HAVE_ACPI_APEI_NMI=y
# CONFIG_ACPI_APEI is not set
# CONFIG_DPTF_POWER is not set
# CONFIG_ACPI_EXTLOG is not set
# CONFIG_PMIC_OPREGION is not set
# CONFIG_ACPI_CONFIGFS is not set
CONFIG_SFI=y

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
# CONFIG_INTEL_IDLE is not set

#
# Memory power savings
#
CONFIG_I7300_IDLE_IOAT_CHANNEL=y
CONFIG_I7300_IDLE=m

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_DOMAINS=y
CONFIG_PCI_CNB20LE_QUIRK=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_PCI_BUS_ADDR_T_64BIT=y
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
CONFIG_PCI_STUB=m
# CONFIG_HT_IRQ is not set
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
# CONFIG_PCI_PRI is not set
CONFIG_PCI_PASID=y
CONFIG_PCI_LABEL=y
# CONFIG_HOTPLUG_PCI is not set

#
# PCI host controller drivers
#
CONFIG_ISA_BUS=y
# CONFIG_ISA_DMA_API is not set
CONFIG_AMD_NB=y
# CONFIG_PCCARD is not set
CONFIG_RAPIDIO=y
CONFIG_RAPIDIO_DISC_TIMEOUT=30
# CONFIG_RAPIDIO_ENABLE_RX_TX_PORTS is not set
CONFIG_RAPIDIO_DMA_ENGINE=y
CONFIG_RAPIDIO_DEBUG=y
CONFIG_RAPIDIO_ENUM_BASIC=m
CONFIG_RAPIDIO_CHMAN=m
CONFIG_RAPIDIO_MPORT_CDEV=y

#
# RapidIO Switch drivers
#
CONFIG_RAPIDIO_TSI57X=y
CONFIG_RAPIDIO_CPS_XX=y
# CONFIG_RAPIDIO_TSI568 is not set
CONFIG_RAPIDIO_CPS_GEN2=y
CONFIG_RAPIDIO_RXS_GEN3=m
# CONFIG_X86_SYSFB is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ELFCORE=y
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
# CONFIG_X86_X32 is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_PMC_ATOM=y
CONFIG_NET=y

#
# Networking options
#
# CONFIG_PACKET is not set
CONFIG_UNIX=y
# CONFIG_UNIX_DIAG is not set
# CONFIG_NET_KEY is not set
# CONFIG_INET is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NET_PTP_CLASSIFY is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
# CONFIG_NETFILTER is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
# CONFIG_BATMAN_ADV is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_DIAG is not set
# CONFIG_MPLS is not set
# CONFIG_HSR is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_SOCK_CGROUP_DATA is not set
# CONFIG_CGROUP_NET_PRIO is not set
# CONFIG_CGROUP_NET_CLASSID is not set
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
# CONFIG_BPF_JIT is not set
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_STREAM_PARSER is not set
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#
CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_NFC is not set
# CONFIG_LWTUNNEL is not set
# CONFIG_DST_CACHE is not set
# CONFIG_NET_DEVLINK is not set
CONFIG_MAY_USE_DEVLINK=y
CONFIG_HAVE_EBPF_JIT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set
CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
CONFIG_DEBUG_DEVRES=y
CONFIG_DEBUG_TEST_DRIVER_REMOVE=y
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
CONFIG_REGMAP_MMIO=y
CONFIG_REGMAP_IRQ=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_FENCE_TRACE is not set
# CONFIG_DMA_CMA is not set

#
# Bus devices
#
# CONFIG_QCOM_EBI2 is not set
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
CONFIG_DTC=y
CONFIG_OF=y
CONFIG_OF_UNITTEST=y
CONFIG_OF_FLATTREE=y
CONFIG_OF_EARLY_FLATTREE=y
CONFIG_OF_DYNAMIC=y
CONFIG_OF_ADDRESS=y
CONFIG_OF_ADDRESS_PCI=y
CONFIG_OF_IRQ=y
CONFIG_OF_PCI=y
CONFIG_OF_PCI_IRQ=y
CONFIG_OF_RESOLVE=y
# CONFIG_OF_OVERLAY is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
# CONFIG_PARPORT_SERIAL is not set
# CONFIG_PARPORT_PC_FIFO is not set
CONFIG_PARPORT_PC_SUPERIO=y
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=m
# CONFIG_PARPORT_1284 is not set
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y

#
# Misc devices
#
CONFIG_SENSORS_LIS3LV02D=y
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
CONFIG_PHANTOM=m
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=y
# CONFIG_TIFM_7XX1 is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
CONFIG_HP_ILO=y
# CONFIG_APDS9802ALS is not set
CONFIG_ISL29003=y
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=y
CONFIG_SENSORS_BH1770=m
# CONFIG_SENSORS_APDS990X is not set
CONFIG_HMC6352=y
# CONFIG_DS1682 is not set
# CONFIG_TI_DAC7512 is not set
# CONFIG_VMWARE_BALLOON is not set
CONFIG_USB_SWITCH_FSA9480=m
CONFIG_LATTICE_ECP3_CONFIG=y
CONFIG_SRAM=y
CONFIG_PANEL=m
CONFIG_PANEL_PARPORT=0
CONFIG_PANEL_PROFILE=5
# CONFIG_PANEL_CHANGE_MESSAGE is not set
CONFIG_C2PORT=m
CONFIG_C2PORT_DURAMAR_2150=m

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_AT25 is not set
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=y
CONFIG_EEPROM_93CX6=y
CONFIG_EEPROM_93XX46=m
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y

#
# Texas Instruments shared transport line discipline
#
CONFIG_SENSORS_LIS3_I2C=y

#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
CONFIG_INTEL_MEI=y
CONFIG_INTEL_MEI_ME=y
CONFIG_INTEL_MEI_TXE=m
CONFIG_VMWARE_VMCI=m

#
# Intel MIC Bus Driver
#
# CONFIG_INTEL_MIC_BUS is not set

#
# SCIF Bus Driver
#
# CONFIG_SCIF_BUS is not set

#
# VOP Bus Driver
#
# CONFIG_VOP_BUS is not set

#
# Intel MIC Host Driver
#

#
# Intel MIC Card Driver
#

#
# SCIF Driver
#

#
# Intel MIC Coprocessor State Management (COSM) Drivers
#

#
# VOP Driver
#
# CONFIG_GENWQE is not set
# CONFIG_ECHO is not set
# CONFIG_CXL_BASE is not set
# CONFIG_CXL_AFU_DRIVER_OPS is not set
CONFIG_HAVE_IDE=y

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_SCSI_DMA is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_NOSY=y
CONFIG_MACINTOSH_DRIVERS=y
# CONFIG_MAC_EMUMOUSEBTN is not set
# CONFIG_NETDEVICES is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_LEDS is not set
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m
CONFIG_INPUT_MATRIXKMAP=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_SAMSUNG is not set
# CONFIG_KEYBOARD_GOLDFISH_EVENTS is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_STMPE is not set
# CONFIG_KEYBOARD_OMAP4 is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_CROS_EC is not set
# CONFIG_KEYBOARD_CAP11XX is not set
# CONFIG_KEYBOARD_BCM is not set
CONFIG_INPUT_MOUSE=y
# CONFIG_MOUSE_PS2 is not set
CONFIG_MOUSE_SERIAL=m
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
CONFIG_MOUSE_CYAPA=m
CONFIG_MOUSE_ELAN_I2C=m
# CONFIG_MOUSE_ELAN_I2C_I2C is not set
CONFIG_MOUSE_ELAN_I2C_SMBUS=y
CONFIG_MOUSE_VSXXXAA=m
CONFIG_MOUSE_SYNAPTICS_I2C=y
# CONFIG_MOUSE_SYNAPTICS_USB is not set
CONFIG_INPUT_JOYSTICK=y
# CONFIG_JOYSTICK_ANALOG is not set
CONFIG_JOYSTICK_A3D=m
# CONFIG_JOYSTICK_ADI is not set
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
# CONFIG_JOYSTICK_GRIP is not set
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=y
CONFIG_JOYSTICK_INTERACT=y
CONFIG_JOYSTICK_SIDEWINDER=y
CONFIG_JOYSTICK_TMDC=y
CONFIG_JOYSTICK_IFORCE=m
# CONFIG_JOYSTICK_IFORCE_232 is not set
CONFIG_JOYSTICK_WARRIOR=y
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
CONFIG_JOYSTICK_ZHENHUA=m
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_AS5011=y
# CONFIG_JOYSTICK_JOYDUMP is not set
# CONFIG_JOYSTICK_XPAD is not set
CONFIG_JOYSTICK_WALKERA0701=m
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_88PM80X_ONKEY=y
CONFIG_INPUT_AD714X=y
CONFIG_INPUT_AD714X_I2C=y
CONFIG_INPUT_AD714X_SPI=m
CONFIG_INPUT_ATMEL_CAPTOUCH=y
CONFIG_INPUT_BMA150=y
# CONFIG_INPUT_E3X0_BUTTON is not set
CONFIG_INPUT_MAX77693_HAPTIC=y
# CONFIG_INPUT_MAX8925_ONKEY is not set
CONFIG_INPUT_MAX8997_HAPTIC=y
# CONFIG_INPUT_MC13783_PWRBUTTON is not set
CONFIG_INPUT_MMA8450=y
CONFIG_INPUT_MPU3050=m
CONFIG_INPUT_APANEL=m
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_KXTJ9 is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_RETU_PWRBUTTON is not set
# CONFIG_INPUT_TPS65218_PWRBUTTON is not set
CONFIG_INPUT_AXP20X_PEK=m
# CONFIG_INPUT_TWL6040_VIBRA is not set
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_PCF8574 is not set
CONFIG_INPUT_PWM_BEEPER=m
CONFIG_INPUT_DA9052_ONKEY=y
# CONFIG_INPUT_DA9063_ONKEY is not set
CONFIG_INPUT_WM831X_ON=m
CONFIG_INPUT_ADXL34X=y
CONFIG_INPUT_ADXL34X_I2C=m
CONFIG_INPUT_ADXL34X_SPI=y
# CONFIG_INPUT_CMA3000 is not set
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
CONFIG_INPUT_DRV2667_HAPTICS=y
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=y
# CONFIG_SERIO_PARKBD is not set
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_SERIO_ARC_PS2 is not set
# CONFIG_SERIO_APBPS2 is not set
CONFIG_USERIO=y
CONFIG_GAMEPORT=y
# CONFIG_GAMEPORT_NS558 is not set
# CONFIG_GAMEPORT_L4 is not set
CONFIG_GAMEPORT_EMU10K1=m
# CONFIG_GAMEPORT_FM801 is not set

#
# Character devices
#
CONFIG_TTY=y
# CONFIG_VT is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_NOZOMI=m
# CONFIG_N_GSM is not set
CONFIG_TRACE_ROUTER=y
CONFIG_TRACE_SINK=y
CONFIG_GOLDFISH_TTY=m
CONFIG_DEVMEM=y
# CONFIG_DEVKMEM is not set

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_FINTEK=y
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_DMA is not set
CONFIG_SERIAL_8250_PCI=m
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
# CONFIG_SERIAL_8250_FSL is not set
CONFIG_SERIAL_8250_DW=m
CONFIG_SERIAL_8250_RT288X=y
# CONFIG_SERIAL_8250_LPSS is not set
# CONFIG_SERIAL_8250_MID is not set
CONFIG_SERIAL_8250_MOXA=y
CONFIG_SERIAL_OF_PLATFORM=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_MAX310X is not set
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=y
# CONFIG_SERIAL_SCCNXP is not set
CONFIG_SERIAL_SC16IS7XX_CORE=m
CONFIG_SERIAL_SC16IS7XX=m
CONFIG_SERIAL_SC16IS7XX_I2C=y
# CONFIG_SERIAL_SC16IS7XX_SPI is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
CONFIG_SERIAL_ALTERA_UART=m
CONFIG_SERIAL_ALTERA_UART_MAXPORTS=4
CONFIG_SERIAL_ALTERA_UART_BAUDRATE=115200
# CONFIG_SERIAL_XILINX_PS_UART is not set
# CONFIG_SERIAL_ARC is not set
CONFIG_SERIAL_RP2=m
CONFIG_SERIAL_RP2_NR_UARTS=32
# CONFIG_SERIAL_FSL_LPUART is not set
CONFIG_SERIAL_CONEXANT_DIGICOLOR=y
# CONFIG_SERIAL_CONEXANT_DIGICOLOR_CONSOLE is not set
CONFIG_SERIAL_MEN_Z135=y
CONFIG_TTY_PRINTK=y
# CONFIG_PRINTER is not set
CONFIG_PPDEV=m
# CONFIG_VIRTIO_CONSOLE is not set
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
# CONFIG_IPMI_SSIF is not set
# CONFIG_IPMI_WATCHDOG is not set
# CONFIG_IPMI_POWEROFF is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_R3964=y
CONFIG_APPLICOM=y
CONFIG_MWAVE=y
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=y
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS_CORE=y
CONFIG_TCG_TIS=y
# CONFIG_TCG_TIS_SPI is not set
CONFIG_TCG_TIS_I2C_ATMEL=y
# CONFIG_TCG_TIS_I2C_INFINEON is not set
CONFIG_TCG_TIS_I2C_NUVOTON=m
CONFIG_TCG_NSC=y
CONFIG_TCG_ATMEL=m
# CONFIG_TCG_INFINEON is not set
# CONFIG_TCG_CRB is not set
CONFIG_TCG_VTPM_PROXY=y
CONFIG_TCG_TIS_ST33ZP24=m
CONFIG_TCG_TIS_ST33ZP24_I2C=m
# CONFIG_TCG_TIS_ST33ZP24_SPI is not set
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
# CONFIG_XILLYBUS is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_ACPI_I2C_OPREGION=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_MUX=y

#
# Multiplexer I2C Chip support
#
CONFIG_I2C_MUX_PCA9541=m
# CONFIG_I2C_MUX_REG is not set
# CONFIG_I2C_HELPER_AUTO is not set
CONFIG_I2C_SMBUS=y

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=y
# CONFIG_I2C_ALGOPCF is not set
CONFIG_I2C_ALGOPCA=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
CONFIG_I2C_ALI1535=y
# CONFIG_I2C_ALI1563 is not set
CONFIG_I2C_ALI15X3=m
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_ISMT=y
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
CONFIG_I2C_SIS5595=y
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=y
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PLATFORM is not set
# CONFIG_I2C_DESIGNWARE_PCI is not set
CONFIG_I2C_EMEV2=m
CONFIG_I2C_KEMPLD=m
CONFIG_I2C_OCORES=y
CONFIG_I2C_PCA_PLATFORM=m
# CONFIG_I2C_PXA_PCI is not set
CONFIG_I2C_RK3X=y
CONFIG_I2C_SIMTEC=y
CONFIG_I2C_XILINX=m

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set

#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_CROS_EC_TUNNEL=m
CONFIG_I2C_STUB=m
CONFIG_I2C_SLAVE=y
CONFIG_I2C_SLAVE_EEPROM=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
CONFIG_SPI_DEBUG=y
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
CONFIG_SPI_ALTERA=y
CONFIG_SPI_AXI_SPI_ENGINE=y
CONFIG_SPI_BITBANG=y
CONFIG_SPI_BUTTERFLY=m
CONFIG_SPI_CADENCE=m
CONFIG_SPI_DESIGNWARE=m
CONFIG_SPI_DW_PCI=m
CONFIG_SPI_DW_MMIO=m
# CONFIG_SPI_LM70_LLP is not set
CONFIG_SPI_FSL_LIB=y
CONFIG_SPI_FSL_SPI=y
CONFIG_SPI_PXA2XX=y
CONFIG_SPI_PXA2XX_PCI=y
CONFIG_SPI_ROCKCHIP=m
CONFIG_SPI_SC18IS602=y
# CONFIG_SPI_XCOMM is not set
# CONFIG_SPI_XILINX is not set
CONFIG_SPI_ZYNQMP_GQSPI=y

#
# SPI Protocol Masters
#
CONFIG_SPI_SPIDEV=m
CONFIG_SPI_LOOPBACK_TEST=m
CONFIG_SPI_TLE62X0=y
# CONFIG_SPMI is not set
CONFIG_HSI=m
CONFIG_HSI_BOARDINFO=y

#
# HSI controllers
#

#
# HSI clients
#
CONFIG_HSI_CHAR=m

#
# PPS support
#
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
CONFIG_NTP_PPS=y

#
# PPS clients support
#
CONFIG_PPS_CLIENT_KTIMER=y
CONFIG_PPS_CLIENT_LDISC=y
CONFIG_PPS_CLIENT_PARPORT=m
CONFIG_PPS_CLIENT_GPIO=y

#
# PPS generators support
#

#
# PTP clock support
#
# CONFIG_PTP_1588_CLOCK is not set

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_GPIOLIB is not set
CONFIG_W1=m

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=m
# CONFIG_W1_MASTER_DS2482 is not set
CONFIG_W1_MASTER_DS1WM=m

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
# CONFIG_W1_SLAVE_DS2408 is not set
CONFIG_W1_SLAVE_DS2413=m
CONFIG_W1_SLAVE_DS2406=m
CONFIG_W1_SLAVE_DS2423=m
CONFIG_W1_SLAVE_DS2431=m
CONFIG_W1_SLAVE_DS2433=m
CONFIG_W1_SLAVE_DS2433_CRC=y
CONFIG_W1_SLAVE_DS2760=m
# CONFIG_W1_SLAVE_DS2780 is not set
CONFIG_W1_SLAVE_DS2781=m
CONFIG_W1_SLAVE_DS28E04=m
CONFIG_W1_SLAVE_BQ27000=m
CONFIG_POWER_AVS=y
# CONFIG_POWER_RESET is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_MAX8925_POWER is not set
# CONFIG_WM831X_BACKUP is not set
CONFIG_WM831X_POWER=y
CONFIG_TEST_POWER=m
CONFIG_BATTERY_ACT8945A=m
CONFIG_BATTERY_DS2760=m
# CONFIG_BATTERY_DS2780 is not set
CONFIG_BATTERY_DS2781=m
CONFIG_BATTERY_DS2782=m
CONFIG_BATTERY_SBS=m
# CONFIG_BATTERY_BQ27XXX is not set
# CONFIG_BATTERY_DA9030 is not set
CONFIG_BATTERY_DA9052=y
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
CONFIG_CHARGER_MAX8903=y
CONFIG_CHARGER_LP8727=y
CONFIG_CHARGER_MAX14577=y
CONFIG_CHARGER_MAX77693=m
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_SMB347 is not set
CONFIG_CHARGER_TPS65090=m
CONFIG_CHARGER_TPS65217=m
CONFIG_BATTERY_GAUGE_LTC2941=m
CONFIG_BATTERY_GOLDFISH=y
CONFIG_BATTERY_RT5033=y
# CONFIG_AXP20X_POWER is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
CONFIG_HWMON_DEBUG_CHIP=y

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
CONFIG_SENSORS_AD7314=m
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=y
CONFIG_SENSORS_ADM1021=y
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
CONFIG_SENSORS_ADT7X10=m
CONFIG_SENSORS_ADT7310=m
CONFIG_SENSORS_ADT7410=m
CONFIG_SENSORS_ADT7411=m
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
CONFIG_SENSORS_K8TEMP=y
CONFIG_SENSORS_K10TEMP=y
# CONFIG_SENSORS_FAM15H_POWER is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_SENSORS_ARM_SCPI is not set
CONFIG_SENSORS_ASB100=y
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS620=y
# CONFIG_SENSORS_DS1621 is not set
CONFIG_SENSORS_DELL_SMM=m
CONFIG_SENSORS_DA9052_ADC=m
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_MC13783_ADC is not set
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_GL518SM=y
# CONFIG_SENSORS_GL520SM is not set
CONFIG_SENSORS_G760A=y
CONFIG_SENSORS_G762=m
# CONFIG_SENSORS_HIH6130 is not set
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_I5500=y
CONFIG_SENSORS_CORETEMP=y
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_JC42 is not set
CONFIG_SENSORS_POWR1220=y
CONFIG_SENSORS_LINEAGE=y
CONFIG_SENSORS_LTC2945=y
# CONFIG_SENSORS_LTC2990 is not set
CONFIG_SENSORS_LTC4151=m
CONFIG_SENSORS_LTC4215=y
CONFIG_SENSORS_LTC4222=m
CONFIG_SENSORS_LTC4245=y
# CONFIG_SENSORS_LTC4260 is not set
CONFIG_SENSORS_LTC4261=m
CONFIG_SENSORS_MAX1111=y
CONFIG_SENSORS_MAX16065=y
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX1668 is not set
# CONFIG_SENSORS_MAX197 is not set
# CONFIG_SENSORS_MAX31722 is not set
# CONFIG_SENSORS_MAX6639 is not set
CONFIG_SENSORS_MAX6642=y
# CONFIG_SENSORS_MAX6650 is not set
CONFIG_SENSORS_MAX6697=m
# CONFIG_SENSORS_MAX31790 is not set
CONFIG_SENSORS_MCP3021=m
CONFIG_SENSORS_MENF21BMC_HWMON=m
# CONFIG_SENSORS_ADCXX is not set
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM70=m
CONFIG_SENSORS_LM73=y
CONFIG_SENSORS_LM75=m
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=y
CONFIG_SENSORS_LM85=y
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
CONFIG_SENSORS_LM92=y
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LM95234 is not set
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_LM95245=m
# CONFIG_SENSORS_PC87360 is not set
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_NTC_THERMISTOR=m
CONFIG_SENSORS_NCT6683=y
CONFIG_SENSORS_NCT6775=m
CONFIG_SENSORS_NCT7802=y
# CONFIG_SENSORS_NCT7904 is not set
# CONFIG_SENSORS_PCF8591 is not set
CONFIG_PMBUS=y
CONFIG_SENSORS_PMBUS=y
# CONFIG_SENSORS_ADM1275 is not set
CONFIG_SENSORS_LM25066=y
CONFIG_SENSORS_LTC2978=m
CONFIG_SENSORS_LTC3815=m
CONFIG_SENSORS_MAX16064=m
CONFIG_SENSORS_MAX20751=y
# CONFIG_SENSORS_MAX34440 is not set
# CONFIG_SENSORS_MAX8688 is not set
CONFIG_SENSORS_TPS40422=m
CONFIG_SENSORS_UCD9000=m
CONFIG_SENSORS_UCD9200=y
# CONFIG_SENSORS_ZL6100 is not set
CONFIG_SENSORS_PWM_FAN=m
CONFIG_SENSORS_SHT21=m
CONFIG_SENSORS_SHT3x=y
# CONFIG_SENSORS_SHTC1 is not set
CONFIG_SENSORS_SIS5595=y
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=y
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_EMC6W201=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=y
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_SCH56XX_COMMON is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_ADC128D818 is not set
CONFIG_SENSORS_ADS1015=y
# CONFIG_SENSORS_ADS7828 is not set
CONFIG_SENSORS_ADS7871=y
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_INA209=m
CONFIG_SENSORS_INA2XX=m
CONFIG_SENSORS_INA3221=y
CONFIG_SENSORS_TC74=m
CONFIG_SENSORS_THMC50=y
CONFIG_SENSORS_TMP102=y
CONFIG_SENSORS_TMP103=y
# CONFIG_SENSORS_TMP401 is not set
CONFIG_SENSORS_TMP421=y
CONFIG_SENSORS_VIA_CPUTEMP=y
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=y
# CONFIG_SENSORS_W83781D is not set
CONFIG_SENSORS_W83791D=m
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
CONFIG_SENSORS_W83795=y
CONFIG_SENSORS_W83795_FANCTRL=y
# CONFIG_SENSORS_W83L785TS is not set
CONFIG_SENSORS_W83L786NG=y
# CONFIG_SENSORS_W83627HF is not set
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_WM831X=m

#
# ACPI drivers
#
# CONFIG_SENSORS_ACPI_POWER is not set
# CONFIG_SENSORS_ATK0110 is not set
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_OF=y
CONFIG_THERMAL_WRITABLE_TRIPS=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
# CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR is not set
CONFIG_THERMAL_GOV_FAIR_SHARE=y
CONFIG_THERMAL_GOV_STEP_WISE=y
# CONFIG_THERMAL_GOV_BANG_BANG is not set
# CONFIG_THERMAL_GOV_USER_SPACE is not set
# CONFIG_THERMAL_GOV_POWER_ALLOCATOR is not set
CONFIG_THERMAL_EMULATION=y
CONFIG_QORIQ_THERMAL=m
CONFIG_INTEL_POWERCLAMP=m
CONFIG_INTEL_SOC_DTS_IOSF_CORE=m
CONFIG_INTEL_SOC_DTS_THERMAL=m

#
# ACPI INT340X thermal drivers
#
# CONFIG_INT340X_THERMAL is not set
CONFIG_INTEL_PCH_THERMAL=m
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_SDIOHOST_POSSIBLE=y
# CONFIG_SSB_SDIOHOST is not set
# CONFIG_SSB_SILENT is not set
CONFIG_SSB_DEBUG=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_BCMA_POSSIBLE=y

#
# Broadcom specific AMBA
#
CONFIG_BCMA=y
CONFIG_BCMA_HOST_PCI_POSSIBLE=y
CONFIG_BCMA_HOST_PCI=y
# CONFIG_BCMA_HOST_SOC is not set
CONFIG_BCMA_DRIVER_PCI=y
# CONFIG_BCMA_DRIVER_GMAC_CMN is not set
CONFIG_BCMA_DEBUG=y

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
CONFIG_MFD_ACT8945A=m
# CONFIG_MFD_AS3711 is not set
CONFIG_MFD_AS3722=m
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_ATMEL_FLEXCOM is not set
CONFIG_MFD_ATMEL_HLCDC=m
CONFIG_MFD_BCM590XX=y
CONFIG_MFD_AXP20X=m
CONFIG_MFD_AXP20X_I2C=m
CONFIG_MFD_CROS_EC=m
# CONFIG_MFD_CROS_EC_I2C is not set
CONFIG_MFD_CROS_EC_SPI=m
CONFIG_PMIC_DA903X=y
CONFIG_PMIC_DA9052=y
CONFIG_MFD_DA9052_SPI=y
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
CONFIG_MFD_DA9062=y
CONFIG_MFD_DA9063=m
# CONFIG_MFD_DA9150 is not set
CONFIG_MFD_EXYNOS_LPASS=y
CONFIG_MFD_MC13XXX=y
CONFIG_MFD_MC13XXX_SPI=y
# CONFIG_MFD_MC13XXX_I2C is not set
CONFIG_MFD_HI6421_PMIC=m
CONFIG_HTC_PASIC3=y
# CONFIG_MFD_INTEL_QUARK_I2C_GPIO is not set
# CONFIG_LPC_ICH is not set
CONFIG_LPC_SCH=m
CONFIG_MFD_INTEL_LPSS=m
# CONFIG_MFD_INTEL_LPSS_ACPI is not set
CONFIG_MFD_INTEL_LPSS_PCI=m
CONFIG_MFD_JANZ_CMODIO=y
CONFIG_MFD_KEMPLD=y
CONFIG_MFD_88PM800=y
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
CONFIG_MFD_MAX14577=y
# CONFIG_MFD_MAX77620 is not set
CONFIG_MFD_MAX77686=y
CONFIG_MFD_MAX77693=y
CONFIG_MFD_MAX77843=y
CONFIG_MFD_MAX8907=y
CONFIG_MFD_MAX8925=y
CONFIG_MFD_MAX8997=y
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6397 is not set
CONFIG_MFD_MENF21BMC=m
# CONFIG_EZX_PCAP is not set
CONFIG_MFD_RETU=y
# CONFIG_MFD_PCF50633 is not set
CONFIG_MFD_RDC321X=y
CONFIG_MFD_RTSX_PCI=m
CONFIG_MFD_RT5033=y
# CONFIG_MFD_RC5T583 is not set
CONFIG_MFD_RK808=m
CONFIG_MFD_RN5T618=y
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
CONFIG_MFD_SM501=m
# CONFIG_MFD_SKY81452 is not set
CONFIG_MFD_SMSC=y
CONFIG_ABX500_CORE=y
# CONFIG_AB3100_CORE is not set
CONFIG_MFD_STMPE=y

#
# STMicroelectronics STMPE Interface Drivers
#
# CONFIG_STMPE_I2C is not set
CONFIG_STMPE_SPI=y
CONFIG_MFD_SYSCON=y
# CONFIG_MFD_TI_AM335X_TSCADC is not set
CONFIG_MFD_LP3943=m
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
CONFIG_TPS6507X=y
CONFIG_MFD_TPS65086=y
CONFIG_MFD_TPS65090=y
CONFIG_MFD_TPS65217=m
# CONFIG_MFD_TI_LP873X is not set
CONFIG_MFD_TPS65218=y
CONFIG_MFD_TPS6586X=y
CONFIG_MFD_TPS65912=y
CONFIG_MFD_TPS65912_I2C=y
CONFIG_MFD_TPS65912_SPI=m
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
CONFIG_TWL6040_CORE=y
CONFIG_MFD_WL1273_CORE=y
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TC3589X is not set
# CONFIG_MFD_TMIO is not set
CONFIG_MFD_VX855=m
CONFIG_MFD_ARIZONA=y
CONFIG_MFD_ARIZONA_I2C=y
CONFIG_MFD_ARIZONA_SPI=y
CONFIG_MFD_CS47L24=y
# CONFIG_MFD_WM5102 is not set
# CONFIG_MFD_WM5110 is not set
CONFIG_MFD_WM8997=y
# CONFIG_MFD_WM8998 is not set
# CONFIG_MFD_WM8400 is not set
CONFIG_MFD_WM831X=y
# CONFIG_MFD_WM831X_I2C is not set
CONFIG_MFD_WM831X_SPI=y
# CONFIG_MFD_WM8350_I2C is not set
CONFIG_MFD_WM8994=y
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=y

#
# Multimedia core support
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
# CONFIG_MEDIA_ANALOG_TV_SUPPORT is not set
# CONFIG_MEDIA_DIGITAL_TV_SUPPORT is not set
# CONFIG_MEDIA_RADIO_SUPPORT is not set
# CONFIG_MEDIA_SDR_SUPPORT is not set
# CONFIG_MEDIA_RC_SUPPORT is not set
CONFIG_MEDIA_CONTROLLER=y
# CONFIG_MEDIA_CONTROLLER_DVB is not set
CONFIG_VIDEO_DEV=y
# CONFIG_VIDEO_V4L2_SUBDEV_API is not set
CONFIG_VIDEO_V4L2=y
CONFIG_VIDEO_ADV_DEBUG=y
CONFIG_VIDEO_FIXED_MINOR_RANGES=y
CONFIG_VIDEO_PCI_SKELETON=y
CONFIG_V4L2_MEM2MEM_DEV=y
CONFIG_VIDEOBUF2_CORE=y
CONFIG_VIDEOBUF2_MEMOPS=y
CONFIG_VIDEOBUF2_DMA_CONTIG=y
CONFIG_VIDEOBUF2_DMA_SG=y
# CONFIG_TTPCI_EEPROM is not set

#
# Media drivers
#
CONFIG_MEDIA_PCI_SUPPORT=y

#
# Media capture support
#
CONFIG_VIDEO_TW5864=y
CONFIG_VIDEO_TW68=y
CONFIG_VIDEO_ZORAN=y
# CONFIG_VIDEO_ZORAN_DC30 is not set
# CONFIG_VIDEO_ZORAN_ZR36060 is not set
CONFIG_V4L_PLATFORM_DRIVERS=y
# CONFIG_VIDEO_CAFE_CCIC is not set
# CONFIG_SOC_CAMERA is not set
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_DEINTERLACE=y
# CONFIG_VIDEO_SH_VEU is not set
# CONFIG_V4L_TEST_DRIVERS is not set

#
# Supported MMC/SDIO adapters
#

#
# Media ancillary drivers (tuners, sensors, i2c, spi, frontends)
#
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y

#
# Audio decoders, processors and mixers
#

#
# RDS decoders
#

#
# Video decoders
#

#
# Video and audio decoders
#

#
# Video encoders
#

#
# Camera sensor devices
#

#
# Flash devices
#

#
# Video improvement chips
#

#
# Audio/Video compression chips
#

#
# Miscellaneous helper chips
#

#
# Sensors used on soc_camera driver
#

#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set

#
# Graphics support
#
CONFIG_AGP=m
CONFIG_AGP_AMD64=m
CONFIG_AGP_INTEL=m
# CONFIG_AGP_SIS is not set
CONFIG_AGP_VIA=m
CONFIG_INTEL_GTT=m
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=m
CONFIG_DRM_MIPI_DSI=y
# CONFIG_DRM_DP_AUX_CHARDEV is not set
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_KMS_FB_HELPER=y
CONFIG_DRM_FBDEV_EMULATION=y
# CONFIG_DRM_LOAD_EDID_FIRMWARE is not set
CONFIG_DRM_TTM=m
CONFIG_DRM_GEM_CMA_HELPER=y
CONFIG_DRM_KMS_CMA_HELPER=y

#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
# CONFIG_DRM_I2C_NXP_TDA998X is not set
CONFIG_DRM_RADEON=m
CONFIG_DRM_RADEON_USERPTR=y
# CONFIG_DRM_AMDGPU is not set

#
# ACP (Audio CoProcessor) Configuration
#
CONFIG_DRM_NOUVEAU=m
CONFIG_NOUVEAU_DEBUG=5
CONFIG_NOUVEAU_DEBUG_DEFAULT=3
# CONFIG_DRM_NOUVEAU_BACKLIGHT is not set
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_PRELIMINARY_HW_SUPPORT is not set
# CONFIG_DRM_I915_USERPTR is not set
CONFIG_DRM_I915_GVT=y

#
# drm/i915 Debugging
#
# CONFIG_DRM_I915_WERROR is not set
CONFIG_DRM_I915_DEBUG=y
CONFIG_DRM_VGEM=m
CONFIG_DRM_VMWGFX=m
# CONFIG_DRM_VMWGFX_FBCON is not set
CONFIG_DRM_GMA500=m
# CONFIG_DRM_GMA600 is not set
CONFIG_DRM_GMA3600=y
# CONFIG_DRM_UDL is not set
CONFIG_DRM_AST=m
CONFIG_DRM_MGAG200=m
# CONFIG_DRM_CIRRUS_QEMU is not set
CONFIG_DRM_QXL=m
CONFIG_DRM_BOCHS=m
CONFIG_DRM_VIRTIO_GPU=m
CONFIG_DRM_PANEL=y

#
# Display Panels
#
# CONFIG_DRM_PANEL_SIMPLE is not set
CONFIG_DRM_PANEL_JDI_LT070ME05000=m
CONFIG_DRM_PANEL_SAMSUNG_LD9040=m
# CONFIG_DRM_PANEL_LG_LG4573 is not set
# CONFIG_DRM_PANEL_PANASONIC_VVX10F034N00 is not set
# CONFIG_DRM_PANEL_SAMSUNG_S6E8AA0 is not set
# CONFIG_DRM_PANEL_SHARP_LQ101R1SX01 is not set
# CONFIG_DRM_PANEL_SHARP_LS043T1LE01 is not set
CONFIG_DRM_BRIDGE=y

#
# Display Interface Bridges
#
CONFIG_DRM_ANALOGIX_ANX78XX=m
CONFIG_DRM_DUMB_VGA_DAC=m
CONFIG_DRM_NXP_PTN3460=m
CONFIG_DRM_PARADE_PS8622=m
# CONFIG_DRM_SII902X is not set
CONFIG_DRM_TOSHIBA_TC358767=m
CONFIG_DRM_I2C_ADV7511=m
CONFIG_DRM_I2C_ADV7533=y
CONFIG_DRM_ARCPGU=m
CONFIG_DRM_LEGACY=y
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_I810=m
CONFIG_DRM_MGA=m
CONFIG_DRM_SIS=m
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m

#
# Frame buffer Devices
#
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB_DDC=y
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
CONFIG_FB_FOREIGN_ENDIAN=y
# CONFIG_FB_BOTH_ENDIAN is not set
CONFIG_FB_BIG_ENDIAN=y
# CONFIG_FB_LITTLE_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_HECUBA=m
CONFIG_FB_SVGALIB=y
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
CONFIG_FB_CYBER2000=m
# CONFIG_FB_CYBER2000_DDC is not set
CONFIG_FB_ARC=m
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
CONFIG_FB_VESA=y
CONFIG_FB_N411=m
CONFIG_FB_HGA=m
CONFIG_FB_OPENCORES=y
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
CONFIG_FB_RIVA=y
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
# CONFIG_FB_RIVA_BACKLIGHT is not set
CONFIG_FB_I740=y
CONFIG_FB_LE80578=y
# CONFIG_FB_CARILLO_RANCH is not set
# CONFIG_FB_INTEL is not set
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
# CONFIG_FB_MATROX_MYSTIQUE is not set
# CONFIG_FB_MATROX_G is not set
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
# CONFIG_FB_RADEON_BACKLIGHT is not set
CONFIG_FB_RADEON_DEBUG=y
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
CONFIG_FB_SAVAGE=y
# CONFIG_FB_SAVAGE_I2C is not set
# CONFIG_FB_SAVAGE_ACCEL is not set
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=y
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
CONFIG_FB_VT8623=y
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
CONFIG_FB_PM3=y
# CONFIG_FB_CARMINE is not set
CONFIG_FB_SM501=m
# CONFIG_FB_IBM_GXT4500 is not set
# CONFIG_FB_GOLDFISH is not set
CONFIG_FB_VIRTUAL=y
CONFIG_FB_METRONOME=y
CONFIG_FB_MB862XX=m
CONFIG_FB_MB862XX_PCI_GDC=y
CONFIG_FB_MB862XX_I2C=y
# CONFIG_FB_BROADSHEET is not set
CONFIG_FB_AUO_K190X=m
CONFIG_FB_AUO_K1900=m
CONFIG_FB_AUO_K1901=m
# CONFIG_FB_SIMPLE is not set
CONFIG_FB_SM712=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=m
# CONFIG_BACKLIGHT_PWM is not set
# CONFIG_BACKLIGHT_DA903X is not set
CONFIG_BACKLIGHT_DA9052=m
CONFIG_BACKLIGHT_MAX8925=m
# CONFIG_BACKLIGHT_APPLE is not set
CONFIG_BACKLIGHT_PM8941_WLED=m
CONFIG_BACKLIGHT_SAHARA=y
# CONFIG_BACKLIGHT_WM831X is not set
CONFIG_BACKLIGHT_ADP8860=y
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3630A is not set
CONFIG_BACKLIGHT_LM3639=m
CONFIG_BACKLIGHT_LP855X=y
CONFIG_BACKLIGHT_TPS65217=m
# CONFIG_BACKLIGHT_LV5207LP is not set
CONFIG_BACKLIGHT_BD6107=m
CONFIG_VGASTATE=y
CONFIG_VIDEOMODE_HELPERS=y
CONFIG_HDMI=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
# CONFIG_SOUND is not set

#
# HID support
#
CONFIG_HID=m
# CONFIG_HID_BATTERY_STRENGTH is not set
CONFIG_HIDRAW=y
CONFIG_UHID=m
CONFIG_HID_GENERIC=m

#
# Special HID drivers
#
# CONFIG_HID_A4TECH is not set
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=m
# CONFIG_HID_ASUS is not set
CONFIG_HID_AUREAL=m
CONFIG_HID_BELKIN=m
# CONFIG_HID_CHERRY is not set
CONFIG_HID_CHICONY=m
CONFIG_HID_CMEDIA=m
# CONFIG_HID_CYPRESS is not set
CONFIG_HID_DRAGONRISE=m
# CONFIG_DRAGONRISE_FF is not set
CONFIG_HID_EMS_FF=m
CONFIG_HID_ELECOM=m
CONFIG_HID_EZKEY=m
# CONFIG_HID_GEMBIRD is not set
CONFIG_HID_GFRM=m
CONFIG_HID_KEYTOUCH=m
CONFIG_HID_KYE=m
CONFIG_HID_WALTOP=m
# CONFIG_HID_GYRATION is not set
# CONFIG_HID_ICADE is not set
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=m
# CONFIG_HID_LCPOWER is not set
CONFIG_HID_LED=m
# CONFIG_HID_LENOVO is not set
CONFIG_HID_LOGITECH=m
# CONFIG_HID_LOGITECH_DJ is not set
CONFIG_HID_LOGITECH_HIDPP=m
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_LOGIG940_FF=y
CONFIG_LOGIWHEELS_FF=y
# CONFIG_HID_MAGICMOUSE is not set
CONFIG_HID_MICROSOFT=m
# CONFIG_HID_MONTEREY is not set
# CONFIG_HID_MULTITOUCH is not set
CONFIG_HID_ORTEK=m
# CONFIG_HID_PANTHERLORD is not set
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
# CONFIG_HID_PICOLCD_FB is not set
CONFIG_HID_PICOLCD_BACKLIGHT=y
# CONFIG_HID_PICOLCD_LEDS is not set
# CONFIG_HID_PLANTRONICS is not set
# CONFIG_HID_PRIMAX is not set
CONFIG_HID_SAITEK=m
# CONFIG_HID_SAMSUNG is not set
# CONFIG_HID_SPEEDLINK is not set
CONFIG_HID_STEELSERIES=m
CONFIG_HID_SUNPLUS=m
CONFIG_HID_RMI=m
# CONFIG_HID_GREENASIA is not set
CONFIG_HID_SMARTJOYPLUS=m
# CONFIG_SMARTJOYPLUS_FF is not set
# CONFIG_HID_TIVO is not set
CONFIG_HID_TOPSEED=m
# CONFIG_HID_THINGM is not set
CONFIG_HID_THRUSTMASTER=m
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_HID_WACOM is not set
CONFIG_HID_WIIMOTE=m
CONFIG_HID_XINMO=m
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
# CONFIG_HID_SENSOR_HUB is not set
CONFIG_HID_ALPS=m

#
# I2C HID support
#
CONFIG_I2C_HID=m

#
# Intel ISH HID support
#
# CONFIG_INTEL_ISH_HID is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
# CONFIG_USB is not set

#
# USB port drivers
#

#
# USB Physical Layer drivers
#
# CONFIG_USB_PHY is not set
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_USB_GADGET is not set
# CONFIG_USB_ULPI_BUS is not set
CONFIG_UWB=m
CONFIG_UWB_WHCI=m
CONFIG_MMC=m
CONFIG_MMC_DEBUG=y
CONFIG_PWRSEQ_EMMC=m
# CONFIG_PWRSEQ_SIMPLE is not set

#
# MMC/SD/SDIO Card Drivers
#
CONFIG_SDIO_UART=m
CONFIG_MMC_TEST=m

#
# MMC/SD/SDIO Host Controller Drivers
#
# CONFIG_MMC_SDHCI is not set
# CONFIG_MMC_TIFM_SD is not set
CONFIG_MMC_GOLDFISH=m
CONFIG_MMC_SPI=m
# CONFIG_MMC_CB710 is not set
CONFIG_MMC_VIA_SDMMC=m
CONFIG_MMC_USDHI6ROL0=m
CONFIG_MMC_REALTEK_PCI=m
CONFIG_MMC_TOSHIBA_PCI=m
# CONFIG_MMC_MTK is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set

#
# MemoryStick drivers
#
CONFIG_MEMSTICK_UNSAFE_RESUME=y

#
# MemoryStick Host Controller Drivers
#
# CONFIG_MEMSTICK_TIFM_MS is not set
# CONFIG_MEMSTICK_JMICRON_38X is not set
CONFIG_MEMSTICK_R592=m
CONFIG_MEMSTICK_REALTEK_PCI=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
# CONFIG_LEDS_CLASS_FLASH is not set

#
# LED drivers
#
CONFIG_LEDS_BCM6328=m
CONFIG_LEDS_BCM6358=y
# CONFIG_LEDS_LM3530 is not set
CONFIG_LEDS_LM3642=m
CONFIG_LEDS_PCA9532=m
CONFIG_LEDS_LP3944=m
CONFIG_LEDS_LP55XX_COMMON=y
CONFIG_LEDS_LP5521=y
# CONFIG_LEDS_LP5523 is not set
# CONFIG_LEDS_LP5562 is not set
# CONFIG_LEDS_LP8501 is not set
# CONFIG_LEDS_LP8860 is not set
CONFIG_LEDS_CLEVO_MAIL=m
CONFIG_LEDS_PCA955X=m
CONFIG_LEDS_PCA963X=y
CONFIG_LEDS_WM831X_STATUS=m
# CONFIG_LEDS_DA903X is not set
CONFIG_LEDS_DA9052=y
CONFIG_LEDS_DAC124S085=m
# CONFIG_LEDS_PWM is not set
CONFIG_LEDS_BD2802=y
# CONFIG_LEDS_INTEL_SS4200 is not set
CONFIG_LEDS_MC13783=y
CONFIG_LEDS_TCA6507=y
CONFIG_LEDS_TLC591XX=y
CONFIG_LEDS_MAX8997=y
CONFIG_LEDS_LM355x=y
# CONFIG_LEDS_MENF21BMC is not set
CONFIG_LEDS_IS31FL319X=y
CONFIG_LEDS_IS31FL32XX=y

#
# LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
#
# CONFIG_LEDS_BLINKM is not set
# CONFIG_LEDS_SYSCON is not set
# CONFIG_LEDS_MLXCPLD is not set

#
# LED Triggers
#
# CONFIG_LEDS_TRIGGERS is not set
# CONFIG_ACCESSIBILITY is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EDAC=y
CONFIG_EDAC_LEGACY_SYSFS=y
CONFIG_EDAC_DEBUG=y
CONFIG_EDAC_DECODE_MCE=m
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_AMD64=m
# CONFIG_EDAC_AMD64_ERROR_INJECTION is not set
# CONFIG_EDAC_E752X is not set
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
CONFIG_EDAC_IE31200=m
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I5000=m
# CONFIG_EDAC_I5100 is not set
CONFIG_EDAC_I7300=m
CONFIG_RTC_LIB=y
CONFIG_RTC_MC146818_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_SYSTOHC is not set
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
# CONFIG_RTC_INTF_SYSFS is not set
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
CONFIG_RTC_INTF_DEV_UIE_EMUL=y
CONFIG_RTC_DRV_TEST=m

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_88PM80X is not set
# CONFIG_RTC_DRV_ABB5ZES3 is not set
CONFIG_RTC_DRV_ABX80X=y
CONFIG_RTC_DRV_AS3722=m
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1307_HWMON is not set
# CONFIG_RTC_DRV_DS1307_CENTURY is not set
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1374_WDT=y
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_HYM8563=y
CONFIG_RTC_DRV_MAX6900=y
# CONFIG_RTC_DRV_MAX8907 is not set
CONFIG_RTC_DRV_MAX8925=m
CONFIG_RTC_DRV_MAX8997=y
# CONFIG_RTC_DRV_MAX77686 is not set
# CONFIG_RTC_DRV_RK808 is not set
CONFIG_RTC_DRV_RS5C372=y
# CONFIG_RTC_DRV_ISL1208 is not set
CONFIG_RTC_DRV_ISL12022=m
CONFIG_RTC_DRV_X1205=m
# CONFIG_RTC_DRV_PCF8523 is not set
CONFIG_RTC_DRV_PCF85063=y
CONFIG_RTC_DRV_PCF8563=m
# CONFIG_RTC_DRV_PCF8583 is not set
CONFIG_RTC_DRV_M41T80=y
# CONFIG_RTC_DRV_M41T80_WDT is not set
CONFIG_RTC_DRV_BQ32K=y
CONFIG_RTC_DRV_TPS6586X=m
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
CONFIG_RTC_DRV_RX8010=y
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
CONFIG_RTC_DRV_EM3027=y
CONFIG_RTC_DRV_RV8803=y

#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_M41T93 is not set
CONFIG_RTC_DRV_M41T94=m
# CONFIG_RTC_DRV_DS1302 is not set
CONFIG_RTC_DRV_DS1305=y
# CONFIG_RTC_DRV_DS1343 is not set
CONFIG_RTC_DRV_DS1347=y
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_MAX6916 is not set
# CONFIG_RTC_DRV_R9701 is not set
CONFIG_RTC_DRV_RX4581=y
CONFIG_RTC_DRV_RX6110=y
CONFIG_RTC_DRV_RS5C348=m
CONFIG_RTC_DRV_MAX6902=m
# CONFIG_RTC_DRV_PCF2123 is not set
CONFIG_RTC_DRV_MCP795=m
CONFIG_RTC_I2C_AND_SPI=y

#
# SPI and I2C RTC drivers
#
# CONFIG_RTC_DRV_DS3232 is not set
CONFIG_RTC_DRV_PCF2127=m
CONFIG_RTC_DRV_RV3029C2=y
CONFIG_RTC_DRV_RV3029_HWMON=y

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=y
# CONFIG_RTC_DRV_DS1511 is not set
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1685_FAMILY=m
# CONFIG_RTC_DRV_DS1685 is not set
# CONFIG_RTC_DRV_DS1689 is not set
# CONFIG_RTC_DRV_DS17285 is not set
# CONFIG_RTC_DRV_DS17485 is not set
CONFIG_RTC_DRV_DS17885=y
# CONFIG_RTC_DS1685_PROC_REGS is not set
# CONFIG_RTC_DS1685_SYSFS_REGS is not set
CONFIG_RTC_DRV_DS1742=y
# CONFIG_RTC_DRV_DS2404 is not set
CONFIG_RTC_DRV_DA9052=y
CONFIG_RTC_DRV_DA9063=y
CONFIG_RTC_DRV_STK17TA8=m
# CONFIG_RTC_DRV_M48T86 is not set
CONFIG_RTC_DRV_M48T35=y
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
# CONFIG_RTC_DRV_BQ4802 is not set
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=y
CONFIG_RTC_DRV_WM831X=y
CONFIG_RTC_DRV_ZYNQMP=m

#
# on-CPU RTC drivers
#
CONFIG_RTC_DRV_MC13XXX=y
CONFIG_RTC_DRV_SNVS=m

#
# HID Sensor RTC drivers
#
CONFIG_DMADEVICES=y
CONFIG_DMADEVICES_DEBUG=y
CONFIG_DMADEVICES_VDEBUG=y

#
# DMA Devices
#
CONFIG_DMA_ENGINE=y
CONFIG_DMA_VIRTUAL_CHANNELS=m
CONFIG_DMA_ACPI=y
CONFIG_DMA_OF=y
CONFIG_FSL_EDMA=m
CONFIG_INTEL_IDMA64=m
CONFIG_INTEL_IOATDMA=m
CONFIG_QCOM_HIDMA_MGMT=y
# CONFIG_QCOM_HIDMA is not set
CONFIG_DW_DMAC_CORE=y
CONFIG_DW_DMAC=y
# CONFIG_DW_DMAC_PCI is not set

#
# DMA Clients
#
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DMA_ENGINE_RAID=y

#
# DMABUF options
#
# CONFIG_SYNC_FILE is not set
CONFIG_DCA=m
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
CONFIG_UIO_CIF=m
# CONFIG_UIO_PDRV_GENIRQ is not set
# CONFIG_UIO_DMEM_GENIRQ is not set
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
# CONFIG_UIO_PCI_GENERIC is not set
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y

#
# Virtio drivers
#
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_VIRTIO_INPUT is not set
CONFIG_VIRTIO_MMIO=y
# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set

#
# Microsoft Hyper-V guest support
#
# CONFIG_HYPERV is not set
CONFIG_STAGING=y
# CONFIG_SLICOSS is not set
CONFIG_COMEDI=m
CONFIG_COMEDI_DEBUG=y
CONFIG_COMEDI_DEFAULT_BUF_SIZE_KB=2048
CONFIG_COMEDI_DEFAULT_BUF_MAXSIZE_KB=20480
CONFIG_COMEDI_MISC_DRIVERS=y
# CONFIG_COMEDI_BOND is not set
CONFIG_COMEDI_TEST=m
CONFIG_COMEDI_PARPORT=m
# CONFIG_COMEDI_SERIAL2002 is not set
CONFIG_COMEDI_ISA_DRIVERS=y
CONFIG_COMEDI_PCL711=m
CONFIG_COMEDI_PCL724=m
CONFIG_COMEDI_PCL726=m
CONFIG_COMEDI_PCL730=m
CONFIG_COMEDI_PCL812=m
CONFIG_COMEDI_PCL816=m
CONFIG_COMEDI_PCL818=m
# CONFIG_COMEDI_PCM3724 is not set
CONFIG_COMEDI_AMPLC_DIO200_ISA=m
CONFIG_COMEDI_AMPLC_PC236_ISA=m
# CONFIG_COMEDI_AMPLC_PC263_ISA is not set
CONFIG_COMEDI_RTI800=m
CONFIG_COMEDI_RTI802=m
# CONFIG_COMEDI_DAC02 is not set
CONFIG_COMEDI_DAS16M1=m
CONFIG_COMEDI_DAS08_ISA=m
CONFIG_COMEDI_DAS16=m
# CONFIG_COMEDI_DAS800 is not set
# CONFIG_COMEDI_DAS1800 is not set
CONFIG_COMEDI_DAS6402=m
# CONFIG_COMEDI_DT2801 is not set
CONFIG_COMEDI_DT2811=m
CONFIG_COMEDI_DT2814=m
# CONFIG_COMEDI_DT2815 is not set
CONFIG_COMEDI_DT2817=m
CONFIG_COMEDI_DT282X=m
CONFIG_COMEDI_DMM32AT=m
CONFIG_COMEDI_FL512=m
CONFIG_COMEDI_AIO_AIO12_8=m
# CONFIG_COMEDI_AIO_IIRO_16 is not set
# CONFIG_COMEDI_II_PCI20KC is not set
CONFIG_COMEDI_C6XDIGIO=m
# CONFIG_COMEDI_MPC624 is not set
# CONFIG_COMEDI_ADQ12B is not set
# CONFIG_COMEDI_NI_AT_A2150 is not set
CONFIG_COMEDI_NI_AT_AO=m
# CONFIG_COMEDI_NI_ATMIO is not set
CONFIG_COMEDI_NI_ATMIO16D=m
# CONFIG_COMEDI_NI_LABPC_ISA is not set
CONFIG_COMEDI_PCMAD=m
CONFIG_COMEDI_PCMDA12=m
CONFIG_COMEDI_PCMMIO=m
CONFIG_COMEDI_PCMUIO=m
CONFIG_COMEDI_MULTIQ3=m
CONFIG_COMEDI_S526=m
CONFIG_COMEDI_PCI_DRIVERS=m
CONFIG_COMEDI_8255_PCI=m
CONFIG_COMEDI_ADDI_WATCHDOG=m
CONFIG_COMEDI_ADDI_APCI_1032=m
# CONFIG_COMEDI_ADDI_APCI_1500 is not set
CONFIG_COMEDI_ADDI_APCI_1516=m
CONFIG_COMEDI_ADDI_APCI_1564=m
CONFIG_COMEDI_ADDI_APCI_16XX=m
CONFIG_COMEDI_ADDI_APCI_2032=m
# CONFIG_COMEDI_ADDI_APCI_2200 is not set
CONFIG_COMEDI_ADDI_APCI_3120=m
CONFIG_COMEDI_ADDI_APCI_3501=m
# CONFIG_COMEDI_ADDI_APCI_3XXX is not set
# CONFIG_COMEDI_ADL_PCI6208 is not set
CONFIG_COMEDI_ADL_PCI7X3X=m
CONFIG_COMEDI_ADL_PCI8164=m
CONFIG_COMEDI_ADL_PCI9111=m
CONFIG_COMEDI_ADL_PCI9118=m
CONFIG_COMEDI_ADV_PCI1710=m
CONFIG_COMEDI_ADV_PCI1720=m
CONFIG_COMEDI_ADV_PCI1723=m
# CONFIG_COMEDI_ADV_PCI1724 is not set
# CONFIG_COMEDI_ADV_PCI1760 is not set
# CONFIG_COMEDI_ADV_PCI_DIO is not set
CONFIG_COMEDI_AMPLC_DIO200_PCI=m
CONFIG_COMEDI_AMPLC_PC236_PCI=m
CONFIG_COMEDI_AMPLC_PC263_PCI=m
# CONFIG_COMEDI_AMPLC_PCI224 is not set
CONFIG_COMEDI_AMPLC_PCI230=m
CONFIG_COMEDI_CONTEC_PCI_DIO=m
CONFIG_COMEDI_DAS08_PCI=m
CONFIG_COMEDI_DT3000=m
# CONFIG_COMEDI_DYNA_PCI10XX is not set
# CONFIG_COMEDI_GSC_HPDI is not set
CONFIG_COMEDI_MF6X4=m
CONFIG_COMEDI_ICP_MULTI=m
CONFIG_COMEDI_DAQBOARD2000=m
CONFIG_COMEDI_JR3_PCI=m
CONFIG_COMEDI_KE_COUNTER=m
CONFIG_COMEDI_CB_PCIDAS64=m
# CONFIG_COMEDI_CB_PCIDAS is not set
CONFIG_COMEDI_CB_PCIDDA=m
CONFIG_COMEDI_CB_PCIMDAS=m
# CONFIG_COMEDI_CB_PCIMDDA is not set
CONFIG_COMEDI_ME4000=m
CONFIG_COMEDI_ME_DAQ=m
# CONFIG_COMEDI_NI_6527 is not set
CONFIG_COMEDI_NI_65XX=m
CONFIG_COMEDI_NI_660X=m
CONFIG_COMEDI_NI_670X=m
# CONFIG_COMEDI_NI_LABPC_PCI is not set
# CONFIG_COMEDI_NI_PCIDIO is not set
CONFIG_COMEDI_NI_PCIMIO=m
# CONFIG_COMEDI_RTD520 is not set
# CONFIG_COMEDI_S626 is not set
CONFIG_COMEDI_MITE=m
CONFIG_COMEDI_NI_TIOCMD=m
CONFIG_COMEDI_8254=m
CONFIG_COMEDI_8255=m
CONFIG_COMEDI_8255_SA=m
# CONFIG_COMEDI_KCOMEDILIB is not set
CONFIG_COMEDI_AMPLC_DIO200=m
CONFIG_COMEDI_AMPLC_PC236=m
CONFIG_COMEDI_DAS08=m
CONFIG_COMEDI_NI_TIO=m
CONFIG_FB_SM750=m
# CONFIG_FB_XGI is not set

#
# Speakup console speech
#
# CONFIG_STAGING_MEDIA is not set

#
# Android
#
# CONFIG_ASHMEM is not set
CONFIG_ANDROID_LOW_MEMORY_KILLER=y
# CONFIG_ION is not set
# CONFIG_STAGING_BOARD is not set
CONFIG_FIREWIRE_SERIAL=m
CONFIG_FWTTY_MAX_TOTAL_PORTS=64
CONFIG_FWTTY_MAX_CARD_PORTS=32
CONFIG_GOLDFISH_AUDIO=m
CONFIG_DGNC=m
CONFIG_GS_FPGABOOT=y
# CONFIG_CRYPTO_SKEIN is not set
# CONFIG_UNISYSSPAR is not set
# CONFIG_COMMON_CLK_XLNX_CLKWZRD is not set
CONFIG_MOST=m
CONFIG_MOSTCORE=m
CONFIG_AIM_CDEV=m
# CONFIG_AIM_NETWORK is not set
# CONFIG_AIM_V4L2 is not set
CONFIG_HDM_I2C=m
# CONFIG_KS7010 is not set
CONFIG_GREYBUS=y
CONFIG_GREYBUS_BOOTROM=m
# CONFIG_GREYBUS_FIRMWARE is not set
# CONFIG_GREYBUS_HID is not set
CONFIG_GREYBUS_LIGHT=m
# CONFIG_GREYBUS_LOG is not set
CONFIG_GREYBUS_LOOPBACK=m
# CONFIG_GREYBUS_POWER is not set
CONFIG_GREYBUS_RAW=y
CONFIG_GREYBUS_VIBRATOR=m
# CONFIG_GREYBUS_BRIDGED_PHY is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACER_WMI is not set
# CONFIG_ACERHDF is not set
# CONFIG_ALIENWARE_WMI is not set
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_DELL_WMI_AIO is not set
# CONFIG_DELL_SMO8800 is not set
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_FUJITSU_TABLET is not set
# CONFIG_HP_ACCEL is not set
# CONFIG_HP_WIRELESS is not set
# CONFIG_HP_WMI is not set
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_ASUS_WIRELESS is not set
CONFIG_ACPI_WMI=m
# CONFIG_MSI_WMI is not set
# CONFIG_TOPSTAR_LAPTOP is not set
# CONFIG_TOSHIBA_BT_RFKILL is not set
# CONFIG_TOSHIBA_HAPS is not set
# CONFIG_TOSHIBA_WMI is not set
# CONFIG_ACPI_CMPC is not set
# CONFIG_INTEL_HID_EVENT is not set
# CONFIG_INTEL_VBTN is not set
# CONFIG_INTEL_IPS is not set
# CONFIG_INTEL_PMC_CORE is not set
# CONFIG_IBM_RTL is not set
# CONFIG_SAMSUNG_LAPTOP is not set
CONFIG_MXM_WMI=m
# CONFIG_SAMSUNG_Q10 is not set
# CONFIG_APPLE_GMUX is not set
# CONFIG_INTEL_RST is not set
# CONFIG_INTEL_SMARTCONNECT is not set
# CONFIG_PVPANIC is not set
# CONFIG_INTEL_PMC_IPC is not set
# CONFIG_SURFACE_PRO3_BUTTON is not set
# CONFIG_INTEL_PUNIT_IPC is not set
CONFIG_GOLDFISH_BUS=y
# CONFIG_GOLDFISH_PIPE is not set
CONFIG_CHROME_PLATFORMS=y
# CONFIG_CHROMEOS_LAPTOP is not set
# CONFIG_CHROMEOS_PSTORE is not set
# CONFIG_CROS_EC_CHARDEV is not set
CONFIG_CROS_EC_LPC=m
CONFIG_CROS_EC_PROTO=y
# CONFIG_CROS_KBD_LED_BACKLIGHT is not set
CONFIG_CLKDEV_LOOKUP=y
CONFIG_HAVE_CLK_PREPARE=y
CONFIG_COMMON_CLK=y

#
# Common Clock Framework
#
CONFIG_COMMON_CLK_WM831X=m
CONFIG_COMMON_CLK_MAX77686=y
CONFIG_COMMON_CLK_RK808=m
# CONFIG_COMMON_CLK_SCPI is not set
CONFIG_COMMON_CLK_SI5351=m
CONFIG_COMMON_CLK_SI514=m
CONFIG_COMMON_CLK_SI570=y
CONFIG_COMMON_CLK_CDCE706=y
# CONFIG_COMMON_CLK_CDCE925 is not set
CONFIG_COMMON_CLK_CS2000_CP=m
CONFIG_CLK_TWL6040=m
# CONFIG_COMMON_CLK_NXP is not set
CONFIG_COMMON_CLK_PWM=m
# CONFIG_COMMON_CLK_PXA is not set
# CONFIG_COMMON_CLK_PIC32 is not set
# CONFIG_COMMON_CLK_MT8135 is not set
# CONFIG_COMMON_CLK_MT8173 is not set

#
# Hardware Spinlock drivers
#

#
# Clock Source drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_CLKBLD_I8253=y
# CONFIG_ATMEL_PIT is not set
# CONFIG_SH_TIMER_CMT is not set
# CONFIG_SH_TIMER_MTU2 is not set
# CONFIG_SH_TIMER_TMU is not set
# CONFIG_EM_TIMER_STI is not set
CONFIG_MAILBOX=y
CONFIG_PLATFORM_MHU=m
# CONFIG_PCC is not set
# CONFIG_ALTERA_MBOX is not set
# CONFIG_MAILBOX_TEST is not set
CONFIG_IOMMU_SUPPORT=y

#
# Generic IOMMU Pagetable Support
#
# CONFIG_AMD_IOMMU is not set

#
# Remoteproc drivers
#
CONFIG_REMOTEPROC=y
CONFIG_STE_MODEM_RPROC=y

#
# Rpmsg drivers
#

#
# SOC (System On Chip) specific Drivers
#

#
# Broadcom SoC drivers
#
# CONFIG_SUNXI_SRAM is not set
# CONFIG_SOC_TI is not set
CONFIG_PM_DEVFREQ=y

#
# DEVFREQ Governors
#
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
# CONFIG_DEVFREQ_GOV_PERFORMANCE is not set
CONFIG_DEVFREQ_GOV_POWERSAVE=y
CONFIG_DEVFREQ_GOV_USERSPACE=m
CONFIG_DEVFREQ_GOV_PASSIVE=y

#
# DEVFREQ Drivers
#
CONFIG_PM_DEVFREQ_EVENT=y
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_NTB is not set
CONFIG_VME_BUS=y

#
# VME Bridge Drivers
#
CONFIG_VME_CA91CX42=y
CONFIG_VME_TSI148=m
CONFIG_VME_FAKE=y

#
# VME Board Drivers
#
# CONFIG_VMIVME_7805 is not set

#
# VME Device Drivers
#
# CONFIG_VME_USER is not set
CONFIG_PWM=y
CONFIG_PWM_SYSFS=y
CONFIG_PWM_ATMEL_HLCDC_PWM=m
CONFIG_PWM_CROS_EC=m
CONFIG_PWM_FSL_FTM=y
CONFIG_PWM_LP3943=m
CONFIG_PWM_LPSS=y
CONFIG_PWM_LPSS_PCI=y
# CONFIG_PWM_LPSS_PLATFORM is not set
CONFIG_PWM_PCA9685=y
# CONFIG_PWM_STMPE is not set
CONFIG_IRQCHIP=y
CONFIG_ARM_GIC_MAX_NR=1
CONFIG_IPACK_BUS=m
# CONFIG_BOARD_TPCI200 is not set
CONFIG_SERIAL_IPOCTAL=m
CONFIG_RESET_CONTROLLER=y
# CONFIG_RESET_ATH79 is not set
# CONFIG_RESET_BERLIN is not set
# CONFIG_RESET_LPC18XX is not set
# CONFIG_RESET_MESON is not set
# CONFIG_RESET_PISTACHIO is not set
# CONFIG_RESET_SOCFPGA is not set
# CONFIG_RESET_STM32 is not set
# CONFIG_RESET_SUNXI is not set
CONFIG_TI_SYSCON_RESET=m
# CONFIG_RESET_ZYNQ is not set
CONFIG_FMC=m
# CONFIG_FMC_FAKEDEV is not set
CONFIG_FMC_TRIVIAL=m
CONFIG_FMC_WRITE_EEPROM=m
# CONFIG_FMC_CHARDEV is not set

#
# PHY Subsystem
#
CONFIG_GENERIC_PHY=y
CONFIG_PHY_PXA_28NM_HSIC=y
CONFIG_PHY_PXA_28NM_USB2=y
# CONFIG_BCM_KONA_USB2_PHY is not set
# CONFIG_POWERCAP is not set
CONFIG_MCB=y
CONFIG_MCB_PCI=m
CONFIG_MCB_LPC=y

#
# Performance monitor support
#
CONFIG_RAS=y
CONFIG_MCE_AMD_INJ=m
CONFIG_THUNDERBOLT=y

#
# Android
#
CONFIG_ANDROID=y
CONFIG_ANDROID_BINDER_IPC=y
CONFIG_NVMEM=m
# CONFIG_STM is not set
# CONFIG_INTEL_TH is not set

#
# FPGA Configuration Support
#
CONFIG_FPGA=y

#
# Firmware Drivers
#
CONFIG_ARM_SCPI_PROTOCOL=m
CONFIG_ARM_SCPI_POWER_DOMAIN=m
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
# CONFIG_FIRMWARE_MEMMAP is not set
CONFIG_DELL_RBU=m
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=y
CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
# CONFIG_ISCSI_IBFT_FIND is not set
CONFIG_FW_CFG_SYSFS=m
# CONFIG_FW_CFG_SYSFS_CMDLINE is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
# CONFIG_MANDATORY_FILE_LOCKING is not set
CONFIG_FSNOTIFY=y
# CONFIG_DNOTIFY is not set
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QFMT_V1=m
# CONFIG_QFMT_V2 is not set
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=m
# CONFIG_CUSE is not set
CONFIG_OVERLAY_FS=y

#
# Caches
#
CONFIG_FSCACHE=m
# CONFIG_FSCACHE_STATS is not set
# CONFIG_FSCACHE_HISTOGRAM is not set
CONFIG_FSCACHE_DEBUG=y
# CONFIG_FSCACHE_OBJECT_LIST is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_PROC_CHILDREN=y
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
CONFIG_ORANGEFS_FS=y
CONFIG_PSTORE=y
CONFIG_PSTORE_ZLIB_COMPRESS=y
# CONFIG_PSTORE_LZO_COMPRESS is not set
# CONFIG_PSTORE_LZ4_COMPRESS is not set
# CONFIG_PSTORE_CONSOLE is not set
CONFIG_PSTORE_PMSG=y
CONFIG_PSTORE_RAM=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
# CONFIG_NLS_CODEPAGE_437 is not set
CONFIG_NLS_CODEPAGE_737=m
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=y
# CONFIG_NLS_CODEPAGE_857 is not set
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=y
# CONFIG_NLS_CODEPAGE_863 is not set
CONFIG_NLS_CODEPAGE_864=y
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
CONFIG_NLS_CODEPAGE_869=y
# CONFIG_NLS_CODEPAGE_936 is not set
CONFIG_NLS_CODEPAGE_950=y
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=y
CONFIG_NLS_CODEPAGE_1250=y
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_MAC_ROMAN=m
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=y
# CONFIG_NLS_MAC_CROATIAN is not set
CONFIG_NLS_MAC_CYRILLIC=y
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=y
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
# CONFIG_NLS_UTF8 is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_DYNAMIC_DEBUG is not set

#
# Compile-time checks and compiler options
#
# CONFIG_DEBUG_INFO is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=8192
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
CONFIG_UNUSED_SYMBOLS=y
# CONFIG_PAGE_OWNER is not set
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_SECTION_MISMATCH=y
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_STACK_VALIDATION=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_DEBUG_KERNEL=y

#
# Memory Debugging
#
CONFIG_PAGE_EXTENSION=y
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_VMACACHE=y
CONFIG_DEBUG_VM_RB=y
CONFIG_DEBUG_VM_PGFLAGS=y
# CONFIG_DEBUG_VIRTUAL is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_KASAN=y
CONFIG_KASAN_OUTLINE=y
# CONFIG_KASAN_INLINE is not set
CONFIG_TEST_KASAN=m
CONFIG_ARCH_HAS_KCOV=y
# CONFIG_KCOV is not set
# CONFIG_DEBUG_SHIRQ is not set

#
# Debug Lockups and Hangs
#
# CONFIG_LOCKUP_DETECTOR is not set
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_WQ_WATCHDOG=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_PANIC_ON_OOPS_VALUE=1
CONFIG_PANIC_TIMEOUT=0
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHED_INFO is not set
# CONFIG_SCHEDSTATS is not set
CONFIG_SCHED_STACK_END_CHECK=y
# CONFIG_DEBUG_TIMEKEEPING is not set
# CONFIG_TIMER_STATS is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y
CONFIG_DEBUG_LOCK_ALLOC=y
# CONFIG_PROVE_LOCKING is not set
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_PI_LIST is not set
# CONFIG_DEBUG_SG is not set
CONFIG_DEBUG_NOTIFIERS=y
# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
# CONFIG_PROVE_RCU is not set
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_TORTURE_TEST=y
CONFIG_RCU_PERF_TEST=y
CONFIG_RCU_TORTURE_TEST=m
# CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT is not set
CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
CONFIG_RCU_TORTURE_TEST_SLOW_INIT_DELAY=3
CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP_DELAY=3
CONFIG_RCU_CPU_STALL_TIMEOUT=21
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_EQS_DEBUG=y
CONFIG_DEBUG_WQ_FORCE_RR_CPU=y
CONFIG_CPU_HOTPLUG_STATE_CONTROL=y
CONFIG_NOTIFIER_ERROR_INJECTION=m
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
# CONFIG_PM_NOTIFIER_ERROR_INJECT is not set
CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT=m
# CONFIG_NETDEV_NOTIFIER_ERROR_INJECT is not set
CONFIG_FAULT_INJECTION=y
# CONFIG_FAILSLAB is not set
CONFIG_FAIL_PAGE_ALLOC=y
CONFIG_FAIL_MMC_REQUEST=y
# CONFIG_FAIL_FUTEX is not set
CONFIG_FAULT_INJECTION_DEBUG_FS=y
# CONFIG_LATENCYTOP is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set

#
# Runtime Testing
#
CONFIG_TEST_LIST_SORT=y
# CONFIG_BACKTRACE_SELF_TEST is not set
CONFIG_RBTREE_TEST=m
CONFIG_INTERVAL_TREE_TEST=m
# CONFIG_PERCPU_TEST is not set
# CONFIG_ATOMIC64_SELFTEST is not set
CONFIG_TEST_HEXDUMP=y
CONFIG_TEST_STRING_HELPERS=y
CONFIG_TEST_KSTRTOX=y
CONFIG_TEST_PRINTF=y
CONFIG_TEST_BITMAP=y
CONFIG_TEST_UUID=y
CONFIG_TEST_RHASHTABLE=m
# CONFIG_TEST_HASH is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_TEST_LKM is not set
CONFIG_TEST_USER_COPY=m
# CONFIG_TEST_BPF is not set
CONFIG_TEST_FIRMWARE=m
# CONFIG_TEST_UDELAY is not set
# CONFIG_MEMTEST is not set
CONFIG_TEST_STATIC_KEYS=m
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_ARCH_WANTS_UBSAN_NO_NULL is not set
CONFIG_UBSAN=y
# CONFIG_UBSAN_SANITIZE_ALL is not set
# CONFIG_UBSAN_ALIGNMENT is not set
CONFIG_UBSAN_NULL=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_X86_PTDUMP_CORE=y
CONFIG_X86_PTDUMP=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_WX is not set
CONFIG_DEBUG_SET_MODULE_RONX=y
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_DOUBLEFAULT=y
CONFIG_DEBUG_TLBFLUSH=y
# CONFIG_IOMMU_DEBUG is not set
CONFIG_IOMMU_STRESS=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set
# CONFIG_DEBUG_ENTRY is not set
# CONFIG_DEBUG_NMI_SELFTEST is not set
CONFIG_X86_DEBUG_FPU=y
CONFIG_PUNIT_ATOM_DEBUG=m

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
CONFIG_SECURITYFS=y
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
CONFIG_HAVE_ARCH_HARDENED_USERCOPY=y
CONFIG_HARDENED_USERCOPY=y
CONFIG_HARDENED_USERCOPY_PAGESPAN=y
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_CRYPTO=m

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_ALGAPI2=m
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=m
CONFIG_CRYPTO_HASH=m
CONFIG_CRYPTO_HASH2=m
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=m
CONFIG_CRYPTO_RNG_DEFAULT=m
CONFIG_CRYPTO_AKCIPHER2=m
CONFIG_CRYPTO_AKCIPHER=m
CONFIG_CRYPTO_KPP2=m
CONFIG_CRYPTO_KPP=m
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_CRYPTO_MANAGER=m
CONFIG_CRYPTO_MANAGER2=m
# CONFIG_CRYPTO_USER is not set
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_NULL2=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=m
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_MCRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_ABLK_HELPER=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
# CONFIG_CRYPTO_GCM is not set
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_SEQIV=m
CONFIG_CRYPTO_ECHAINIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
CONFIG_CRYPTO_KEYWRAP=m

#
# Hash modes
#
# CONFIG_CRYPTO_CMAC is not set
CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m

#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_CRC32_PCLMUL=m
CONFIG_CRYPTO_CRCT10DIF=m
CONFIG_CRYPTO_CRCT10DIF_PCLMUL=m
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_POLY1305=m
# CONFIG_CRYPTO_POLY1305_X86_64 is not set
CONFIG_CRYPTO_MD4=m
# CONFIG_CRYPTO_MD5 is not set
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
# CONFIG_CRYPTO_SHA1_MB is not set
CONFIG_CRYPTO_SHA256_MB=m
CONFIG_CRYPTO_SHA512_MB=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
# CONFIG_CRYPTO_SHA3 is not set
# CONFIG_CRYPTO_TGR192 is not set
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_CAST6_AVX_X86_64 is not set
CONFIG_CRYPTO_DES=m
# CONFIG_CRYPTO_DES3_EDE_X86_64 is not set
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
CONFIG_CRYPTO_CHACHA20=m
# CONFIG_CRYPTO_CHACHA20_X86_64 is not set
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
CONFIG_CRYPTO_LZ4HC=m

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_DRBG_MENU=m
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
CONFIG_CRYPTO_DRBG_CTR=y
CONFIG_CRYPTO_DRBG=m
CONFIG_CRYPTO_JITTERENTROPY=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
# CONFIG_CRYPTO_DEV_CCP is not set
CONFIG_CRYPTO_DEV_QAT=m
CONFIG_CRYPTO_DEV_QAT_DH895xCC=m
# CONFIG_CRYPTO_DEV_QAT_C3XXX is not set
# CONFIG_CRYPTO_DEV_QAT_C62X is not set
CONFIG_CRYPTO_DEV_QAT_DH895xCCVF=m
CONFIG_CRYPTO_DEV_QAT_C3XXXVF=m
# CONFIG_CRYPTO_DEV_QAT_C62XVF is not set

#
# Certificates for signature checking
#
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
# CONFIG_VHOST_NET is not set
CONFIG_VHOST_CROSS_ENDIAN_LEGACY=y
# CONFIG_BINARY_PRINTF is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
# CONFIG_HAVE_ARCH_BITREVERSE is not set
CONFIG_RATIONAL=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
# CONFIG_CRC_CCITT is not set
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC32_SELFTEST=y
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
CONFIG_CRC7=m
# CONFIG_LIBCRC32C is not set
CONFIG_CRC8=y
# CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4HC_COMPRESS=m
CONFIG_LZ4_DECOMPRESS=m
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
# CONFIG_XZ_DEC_IA64 is not set
# CONFIG_XZ_DEC_ARM is not set
# CONFIG_XZ_DEC_ARMTHUMB is not set
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_ENC8=y
CONFIG_REED_SOLOMON_DEC8=y
CONFIG_INTERVAL_TREE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_CLZ_TAB=y
# CONFIG_CORDIC is not set
# CONFIG_DDR is not set
CONFIG_IRQ_POLL=y
CONFIG_MPILIB=m
CONFIG_LIBFDT=y
# CONFIG_SG_SPLIT is not set
# CONFIG_SG_POOL is not set
CONFIG_ARCH_HAS_SG_CHAIN=y
CONFIG_ARCH_HAS_PMEM_API=y
CONFIG_ARCH_HAS_MMIO_FLUSH=y
CONFIG_STACKDEPOT=y

[-- Attachment #3: job-script --]
[-- Type: text/plain, Size: 4040 bytes --]

#!/bin/sh

export_top_env()
{
	export suite='boot'
	export testcase='boot'
	export timeout='10m'
	export job_origin='/lkp/lkp/src/jobs/boot.yaml'
	export queue='bisect'
	export testbox='vm-lkp-wildcat-yocto-ia32-5'
	export tbox_group='vm-lkp-wildcat-yocto-ia32'
	export branch='linux-devel/devel-catchup-201611182101'
	export commit='83ee00c6cf5eaa85f74094d6800732edf7114ef9'
	export kconfig='x86_64-randconfig-s3-11182101'
	export submit_id='582f28f60b9a93572cf71a57'
	export job_file='/lkp/scheduled/vm-lkp-wildcat-yocto-ia32-5/boot-1-yocto-tiny-i386-2016-04-22.cgz-83ee00c6cf5eaa85f74094d6800732edf7114ef9-20161119-87852-nnwp4h-0.yaml'
	export id='4835cbf0f4efa7bcd6d9a7450986217fca1a46cc'
	export model='qemu-system-x86_64 -enable-kvm'
	export nr_vm=8
	export nr_cpu=1
	export memory='320M'
	export rootfs='yocto-tiny-i386-2016-04-22.cgz'
	export need_kconfig='CONFIG_KVM_GUEST=y'
	export compiler='gcc-6'
	export enqueue_time='2016-11-19 00:14:46 +0800'
	export _id='582f28f60b9a93572cf71a57'
	export user='lkp'
	export result_root='/result/boot/1/vm-lkp-wildcat-yocto-ia32/yocto-tiny-i386-2016-04-22.cgz/x86_64-randconfig-s3-11182101/gcc-6/83ee00c6cf5eaa85f74094d6800732edf7114ef9/0'
	export LKP_SERVER='inn'
	export max_uptime=600
	export initrd='/osimage/yocto/yocto-tiny-i386-2016-04-22.cgz'
	export bootloader_append='root=/dev/ram0
user=lkp
job=/lkp/scheduled/vm-lkp-wildcat-yocto-ia32-5/boot-1-yocto-tiny-i386-2016-04-22.cgz-83ee00c6cf5eaa85f74094d6800732edf7114ef9-20161119-87852-nnwp4h-0.yaml
ARCH=x86_64
kconfig=x86_64-randconfig-s3-11182101
branch=linux-devel/devel-catchup-201611182101
commit=83ee00c6cf5eaa85f74094d6800732edf7114ef9
BOOT_IMAGE=/pkg/linux/x86_64-randconfig-s3-11182101/gcc-6/83ee00c6cf5eaa85f74094d6800732edf7114ef9/vmlinuz-4.9.0-rc1-00041-g83ee00c
max_uptime=600
RESULT_ROOT=/result/boot/1/vm-lkp-wildcat-yocto-ia32/yocto-tiny-i386-2016-04-22.cgz/x86_64-randconfig-s3-11182101/gcc-6/83ee00c6cf5eaa85f74094d6800732edf7114ef9/0
LKP_SERVER=inn
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0
printk.devkmsg=on
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
systemd.log_level=err
ignore_loglevel
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
rw'
	export lkp_initrd='/lkp/lkp/lkp-i386.cgz'
	export modules_initrd='/pkg/linux/x86_64-randconfig-s3-11182101/gcc-6/83ee00c6cf5eaa85f74094d6800732edf7114ef9/modules.cgz'
	export bm_initrd='/osimage/deps/debian-x86_64-2016-08-31.cgz/run-ipconfig.i386_2016-09-03.cgz'
	export site='inn'
	export LKP_CGI_PORT=80
	export LKP_CIFS_PORT=139
	export kernel='/pkg/linux/x86_64-randconfig-s3-11182101/gcc-6/83ee00c6cf5eaa85f74094d6800732edf7114ef9/vmlinuz-4.9.0-rc1-00041-g83ee00c'
	export dequeue_time='2016-11-19 00:35:22 +0800'
	export job_initrd='/lkp/scheduled/vm-lkp-wildcat-yocto-ia32-5/boot-1-yocto-tiny-i386-2016-04-22.cgz-83ee00c6cf5eaa85f74094d6800732edf7114ef9-20161119-87852-nnwp4h-0.cgz'

	[ -n "$LKP_SRC" ] ||
	export LKP_SRC=/lkp/${user:-lkp}/src
}

run_job()
{
	echo $$ > $TMP/run-job.pid

	. $LKP_SRC/lib/http.sh
	. $LKP_SRC/lib/job.sh
	. $LKP_SRC/lib/env.sh

	export_top_env

	run_monitor $LKP_SRC/monitors/one-shot/wrapper boot-slabinfo
	run_monitor $LKP_SRC/monitors/one-shot/wrapper boot-meminfo
	run_monitor $LKP_SRC/monitors/one-shot/wrapper memmap
	run_monitor $LKP_SRC/monitors/wrapper kmsg
	run_monitor $LKP_SRC/monitors/wrapper oom-killer
	run_monitor $LKP_SRC/monitors/plain/watchdog
	run_monitor $LKP_SRC/monitors/wrapper nfs-hang

	run_test $LKP_SRC/tests/wrapper sleep 1
}

extract_stats()
{
	$LKP_SRC/stats/wrapper boot-slabinfo
	$LKP_SRC/stats/wrapper boot-meminfo
	$LKP_SRC/stats/wrapper memmap
	$LKP_SRC/stats/wrapper boot-memory
	$LKP_SRC/stats/wrapper boot-time
	$LKP_SRC/stats/wrapper kernel-size
	$LKP_SRC/stats/wrapper kmsg

	$LKP_SRC/stats/wrapper time sleep.time
	$LKP_SRC/stats/wrapper time
	$LKP_SRC/stats/wrapper dmesg
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper stderr
	$LKP_SRC/stats/wrapper last_state
}

"$@"

[-- Attachment #4: dmesg.xz --]
[-- Type: application/octet-stream, Size: 10384 bytes --]