kernel panic in 4.2.3, rb_erase in sch_fq

kernel panic in 4.2.3, rb_erase in sch_fq

[Denys Fedoryshchenko]

Hi!

Actually seems i was getting this panic for a while (once per week) on 
loaded pppoe server, but just now was able to get full panic message.
After checking commit logs on sch_fq.c i didnt seen any fixes, so 
probably upgrading to newer kernel wont help?


<remote> [237470.633382] general protection fault: 0000 [#1]
<remote> SMP
<remote>
<remote> [237470.633832] Modules linked in:
<remote> netconsole
<remote> configfs
<remote> act_skbedit
<remote> sch_fq
<remote> cls_fw
<remote> act_police
<remote> cls_u32
<remote> sch_ingress
<remote> sch_sfq
<remote> sch_htb
<remote> pppoe
<remote> pppox
<remote> ppp_generic
<remote> slhc
<remote> xt_nat
<remote> ts_bm
<remote> xt_string
<remote> xt_connmark
<remote> xt_TCPMSS
<remote> xt_tcpudp
<remote> xt_mark
<remote> iptable_filter
<remote> iptable_nat
<remote> nf_conntrack_ipv4
<remote> nf_defrag_ipv4
<remote> nf_nat_ipv4
<remote> nf_nat
<remote> nf_conntrack
<remote> iptable_mangle
<remote> ip_tables
<remote> x_tables
<remote> 8021q
<remote> garp
<remote> mrp
<remote> stp
<remote> llc
<remote>
<remote> [237470.637835] CPU: 1 PID: 14035 Comm: accel-pppd Not tainted 
4.2.3-build-0087 #3
<remote> [237470.638342] Hardware name: Intel Corporation 
S2600GZ/S2600GZ, BIOS SE5C600.86B.01.03.0002.062020121504 06/20/2012
<remote> [237470.638859] task: ffff8803ef8b5080 ti: ffff8803ed7e0000 
task.ti: ffff8803ed7e0000
<remote> [237470.639370] RIP: 0010:[<ffffffff8125af5d>]
<remote> [<ffffffff8125af5d>] rb_erase+0x37/0x2c4
<remote> [237470.639960] RSP: 0018:ffff8803ed7e3b88  EFLAGS: 00010286
<remote> [237470.644863] RAX: 0000000000000000 RBX: ffff8804106ab000 
RCX: 0000000000000001
<remote> [237470.645366] RDX: ffa2050402210218 RSI: ffff88040cfe2cf0 
RDI: ffff8803f50d00e0
<remote> [237470.645872] RBP: ffff8803ed7e3b88 R08: 0000000000000000 
R09: ffff88042ee37d50
<remote> [237470.646376] R10: ffffea000fe7a9c0 R11: 0000000094f1b850 
R12: 000000000000019e
<remote> [237470.646881] R13: ffff88040cfe2cf0 R14: ffff8803f50d00d0 
R15: 0000000000000000
<remote> [237470.647381] FS:  00007fcd5d384700(0000) 
GS:ffff88042ee20000(0000) knlGS:0000000000000000
<remote> [237470.647889] CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
<remote> [237470.648209] CR2: 00007fcd003efa90 CR3: 0000000424b6e000 
CR4: 00000000000406e0
<remote> [237470.648707] Stack:
<remote> [237470.648990]  ffff8803ed7e3bb8
<remote> ffffffffa00ef38b
<remote> ffff8804106ab000
<remote> ffff880416079000
<remote>
<remote> [237470.649791]  0000000000000002
<remote> ffff8804160790d8
<remote> ffff8803ed7e3bd8
<remote> ffffffff8183785c
<remote>
<remote> [237470.650589]  0000000000000002
<remote> ffff8800b021d000
<remote> ffff8803ed7e3c18
<remote> ffffffffa00d247a
<remote>
<remote> [237470.651387] Call Trace:
<remote> [237470.651716]  [<ffffffffa00ef38b>] fq_reset+0x7a/0xf2 
[sch_fq]
<remote> [237470.652084]  [<ffffffff8183785c>] qdisc_reset+0x18/0x42
<remote> [237470.652444]  [<ffffffffa00d247a>] htb_reset+0x96/0x14d 
[sch_htb]
<remote> [237470.652780]  [<ffffffff8183785c>] qdisc_reset+0x18/0x42
<remote> [237470.653146]  [<ffffffff81837e3d>] 
dev_deactivate_queue.constprop.34+0x43/0x53
<remote> [237470.653726]  [<ffffffff81838543>] 
dev_deactivate_many+0x53/0x206
<remote> [237470.654088]  [<ffffffff81819ce2>] 
__dev_close_many+0x73/0xbf
<remote> [237470.654436]  [<ffffffff81819e5d>] __dev_close+0x2c/0x41
<remote> [237470.654784]  [<ffffffff818a5569>] ? 
_raw_spin_unlock_bh+0x15/0x17
<remote> [237470.655106]  [<ffffffff8181ff25>] 
__dev_change_flags+0xa5/0x13c
<remote> [237470.655427]  [<ffffffff8181ffdf>] 
dev_change_flags+0x23/0x59
<remote> [237470.655777]  [<ffffffff818a437b>] ? mutex_lock+0x13/0x24
<remote> [237470.656073]  [<ffffffff81874197>] devinet_ioctl+0x246/0x533
<remote> [237470.656372]  [<ffffffff8187572d>] inet_ioctl+0x8c/0xa6
<remote> [237470.656667]  [<ffffffff81807e06>] sock_do_ioctl+0x22/0x40
<remote> [237470.656960]  [<ffffffff81808016>] sock_ioctl+0x1f2/0x200
<remote> [237470.657253]  [<ffffffff8115f5b5>] do_vfs_ioctl+0x360/0x41a
<remote> [237470.657549]  [<ffffffff81151989>] ? vfs_write+0x105/0x164
<remote> [237470.657841]  [<ffffffff8115f6a8>] SyS_ioctl+0x39/0x61
<remote> [237470.658134]  [<ffffffff818a591b>] 
entry_SYSCALL_64_fastpath+0x16/0x6e
<remote> [237470.658431] Code:
<remote> 48
<remote> 85
<remote> c0
<remote> 75
<remote> 36
<remote> 48
<remote> 8b
<remote> 0f
<remote> 48
<remote> 89
<remote> c8
<remote> 48
<remote> 83
<remote> e0
<remote> fc
<remote> 74
<remote> 12
<remote> 48
<remote> 39
<remote> 78
<remote> 10
<remote> 75
<remote> 06
<remote> 48
<remote> 89
<remote> 50
<remote> 10
<remote> eb
<remote> 09
<remote> 48
<remote> 89
<remote> 50
<remote> 08
<remote> eb
<remote> 03
<remote> 48
<remote> 89
<remote> 16
<remote> 48
<remote> 85
<remote> d2
<remote> 74
<remote> 08
<remote>
<remote> 89
<remote> 0a
<remote> e9
<remote> 83
<remote> 02
<remote> 00
<remote> 00
<remote> 80
<remote> e1
<remote> 01
<remote> e9
<remote> c3
<remote> 00
<remote> 00
<remote> 00
<remote> 48
<remote> 85
<remote> d2
<remote> 75
<remote> 2c
<remote>
<remote> [237470.663930] RIP
<remote> [<ffffffff8125af5d>] rb_erase+0x37/0x2c4
<remote> [237470.664296]  RSP <ffff8803ed7e3b88>
<remote> [237470.664598] ---[ end trace 32ea40a7de450892 ]---
<remote> [237470.673272] Kernel panic - not syncing: Fatal exception in 
interrupt
<remote> [237470.673577] Kernel Offset: disabled
<remote> [237470.704654] Rebooting in 5 seconds..

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Eric Dumazet]

On Mon, 2015-11-02 at 16:11 +0200, Denys Fedoryshchenko wrote:
> Hi!
> 
> Actually seems i was getting this panic for a while (once per week) on 
> loaded pppoe server, but just now was able to get full panic message.
> After checking commit logs on sch_fq.c i didnt seen any fixes, so 
> probably upgrading to newer kernel wont help?

I do not think we support sch_fq as a HTB leaf.

If you want both HTB and sch_fq, you need to setup a bonding device.

HTB on bond0

sch_fq on the slaves

Sure, the kernel should not crash, but HTB+sch_fq on same net device is
certainly not something that will work anyway.

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Denys Fedoryshchenko]

On 2015-11-02 17:24, Eric Dumazet wrote:
> On Mon, 2015-11-02 at 16:11 +0200, Denys Fedoryshchenko wrote:
>> Hi!
>> 
>> Actually seems i was getting this panic for a while (once per week) on
>> loaded pppoe server, but just now was able to get full panic message.
>> After checking commit logs on sch_fq.c i didnt seen any fixes, so
>> probably upgrading to newer kernel wont help?
> 
> I do not think we support sch_fq as a HTB leaf.
> 
> If you want both HTB and sch_fq, you need to setup a bonding device.
> 
> HTB on bond0
> 
> sch_fq on the slaves
> 
> Sure, the kernel should not crash, but HTB+sch_fq on same net device is
> certainly not something that will work anyway.
Strange, because except ppp, on static devices it works really very well 
in such scheme. It is the only solution that can throttle incoming 
bandwidth, when bandwidth is very overbooked - reliably, for my use 
cases, such as 256k+ flows/2.5Gbps and several different classes of 
traffic, so using DRR will end up in just not enough classes.

On latest kernels i had to patch tc to provide parameter for orphan mask 
in fq, to increase number for flows for transit traffic.
None of other qdiscs able to solve this problem, incoming bandwidth 
simply flowing 10-20% more than set, but fq is doing magic.
The only device that was working with similar efficiency for such cases 
- proprietary PacketShaper, but is modifying tcp window size, and can't 
be called transparent, and also has stability issues over 1Gbps.

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Eric Dumazet]

On Mon, 2015-11-02 at 17:58 +0200, Denys Fedoryshchenko wrote:
> On 2015-11-02 17:24, Eric Dumazet wrote:
> > On Mon, 2015-11-02 at 16:11 +0200, Denys Fedoryshchenko wrote:
> >> Hi!
> >> 
> >> Actually seems i was getting this panic for a while (once per week) on
> >> loaded pppoe server, but just now was able to get full panic message.
> >> After checking commit logs on sch_fq.c i didnt seen any fixes, so
> >> probably upgrading to newer kernel wont help?
> > 
> > I do not think we support sch_fq as a HTB leaf.
> > 
> > If you want both HTB and sch_fq, you need to setup a bonding device.
> > 
> > HTB on bond0
> > 
> > sch_fq on the slaves
> > 
> > Sure, the kernel should not crash, but HTB+sch_fq on same net device is
> > certainly not something that will work anyway.
> Strange, because except ppp, on static devices it works really very well 
> in such scheme. It is the only solution that can throttle incoming 
> bandwidth, when bandwidth is very overbooked - reliably, for my use 
> cases, such as 256k+ flows/2.5Gbps and several different classes of 
> traffic, so using DRR will end up in just not enough classes.
> 
> On latest kernels i had to patch tc to provide parameter for orphan mask 
> in fq, to increase number for flows for transit traffic.
> None of other qdiscs able to solve this problem, incoming bandwidth 
> simply flowing 10-20% more than set, but fq is doing magic.
> The only device that was working with similar efficiency for such cases 
> - proprietary PacketShaper, but is modifying tcp window size, and can't 
> be called transparent, and also has stability issues over 1Gbps.

Ah, I was thinking you needed more like 10Gb traffic ;)

with HTB on bonding, we can use MQ+FQ on the slaves in order to use many
cpus to serve local traffic.

But yes, if you use HTB+FQ for forwarding, I guess the bonding setup is
not really needed.

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Denys Fedoryshchenko]

On 2015-11-02 18:12, Eric Dumazet wrote:
> On Mon, 2015-11-02 at 17:58 +0200, Denys Fedoryshchenko wrote:
>> On 2015-11-02 17:24, Eric Dumazet wrote:
>> > On Mon, 2015-11-02 at 16:11 +0200, Denys Fedoryshchenko wrote:
>> >> Hi!
>> >>
>> >> Actually seems i was getting this panic for a while (once per week) on
>> >> loaded pppoe server, but just now was able to get full panic message.
>> >> After checking commit logs on sch_fq.c i didnt seen any fixes, so
>> >> probably upgrading to newer kernel wont help?
>> >
>> > I do not think we support sch_fq as a HTB leaf.
>> >
>> > If you want both HTB and sch_fq, you need to setup a bonding device.
>> >
>> > HTB on bond0
>> >
>> > sch_fq on the slaves
>> >
>> > Sure, the kernel should not crash, but HTB+sch_fq on same net device is
>> > certainly not something that will work anyway.
>> Strange, because except ppp, on static devices it works really very 
>> well
>> in such scheme. It is the only solution that can throttle incoming
>> bandwidth, when bandwidth is very overbooked - reliably, for my use
>> cases, such as 256k+ flows/2.5Gbps and several different classes of
>> traffic, so using DRR will end up in just not enough classes.
>> 
>> On latest kernels i had to patch tc to provide parameter for orphan 
>> mask
>> in fq, to increase number for flows for transit traffic.
>> None of other qdiscs able to solve this problem, incoming bandwidth
>> simply flowing 10-20% more than set, but fq is doing magic.
>> The only device that was working with similar efficiency for such 
>> cases
>> - proprietary PacketShaper, but is modifying tcp window size, and 
>> can't
>> be called transparent, and also has stability issues over 1Gbps.
> 
> Ah, I was thinking you needed more like 10Gb traffic ;)
> 
> with HTB on bonding, we can use MQ+FQ on the slaves in order to use 
> many
> cpus to serve local traffic.
> 
> But yes, if you use HTB+FQ for forwarding, I guess the bonding setup is
> not really needed.
Well, here country is very underdeveloped in matters of technology. 10G 
interfaces appeared in some ISP only this year.
On the ppp interfaces where crash happening - it is even less bandwidth. 
Each user max 1-2Mbps(average usage 128kbps), 4.5k interfaces.
But i have some more heavy setups there, around 9k pppoe users 
terminated on single server, (means 9k interfaces), about 2Gbps traffic 
passing thru.
If i take non-FOSS solution, i will have to pay for software licenses 
$100k+, which is unbearable for local ISP. fq is not critical in this 
specific use case, i can use for ppp interfaces fifo or such, but i 
guess better to report a but :)

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Cong Wang]

On Mon, Nov 2, 2015 at 6:11 AM, Denys Fedoryshchenko
<nuclearcat@nuclearcat.com> wrote:
> Hi!
>
> Actually seems i was getting this panic for a while (once per week) on
> loaded pppoe server, but just now was able to get full panic message.
> After checking commit logs on sch_fq.c i didnt seen any fixes, so probably
> upgrading to newer kernel wont help?


Can you share your `tc qdisc show dev xxxx` with us? And how to reproduce
it? I tried to setup htb+fq and then flip the interface back and forth
but I don't
see any crash.

Thanks.

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Denys Fedoryshchenko]

On 2015-11-04 00:06, Cong Wang wrote:
> On Mon, Nov 2, 2015 at 6:11 AM, Denys Fedoryshchenko
> <nuclearcat@nuclearcat.com> wrote:
>> Hi!
>> 
>> Actually seems i was getting this panic for a while (once per week) on
>> loaded pppoe server, but just now was able to get full panic message.
>> After checking commit logs on sch_fq.c i didnt seen any fixes, so 
>> probably
>> upgrading to newer kernel wont help?
> 
> 
> Can you share your `tc qdisc show dev xxxx` with us? And how to 
> reproduce
> it? I tried to setup htb+fq and then flip the interface back and forth
> but I don't
> see any crash.
My guess it wont be easy to reproduce, it is happening on box with 4.5k 
interfaces, that constantly create/delete interfaces,
and even with that this problem may happen once per day, or may not 
happen for 1 week.

Here is script that is being fired after new ppp interface detected. But 
pppoe process are independent from
process that are "establishing" shapers.

/sbin/tc qdisc del  root
/sbin/tc qdisc add  handle 1: root htb default 3

/sbin/tc filter add parent 1:0 protocol ip prio 4 handle 1 fw flowid 1:3
/sbin/tc filter add parent 1:0 protocol ip prio 3 u32 match ip protocol 
6 0xff match ip src 10.0.252.8/32 flowid 1:3/sbin/tc filter add parent 
1:0 protocol ip prio 5 u32 match ip protocol 1 0xff flowid 1:0
/sbin/tc filter add parent 1:0 protocol ip prio 5 u32 match ip protocol 
6 0xff match ip sport 80 0xffff flowid 1:4
/sbin/tc filter add parent 1:0 protocol ip prio 5 u32 match ip protocol 
6 0xff match ip sport 443 0xffff flowid 1:5
/sbin/tc filter add parent 1:0 protocol ip prio 100 u32 match u32 0 0 
flowid 1:2

/sbin/tc class add  classid 1:1 parent 1:0 htb rate 512Kbit ceil 
512Kbit.
/sbin/tc class add  classid 1:2 parent 1:1 htb rate 32Kbit ceil 512Kbit
/sbin/tc class add  classid 1:3 parent 1:0 htb rate 10Mbit ceil 10Mbit
/sbin/tc class add  classid 1:4 parent 1:1 htb rate 32Kbit ceil 512Kbit
/sbin/tc class add  classid 1:5 parent 1:1 htb rate 32Kbit ceil 512Kbit

/sbin/tc qdisc add parent 1:2 fq limit 300
/sbin/tc qdisc add parent 1:3 pfifo limit 300
/sbin/tc qdisc add parent 1:4 fq limit 300
/sbin/tc qdisc add parent 1:5 fq limit 300

Possible cases come to my mind (but maybe i missed others):
  Script and tc working and interface are deleted in a process (e.g. 
interface disappears)
  Script deleting root while there is heavy traffic on interface and a 
lot of packets queued
  ppp interface destroyed, while there is a lot of traffic queued on it 
(this one a bit rare situation)

> 
> Thanks.

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Eric Dumazet]

On Wed, 2015-11-04 at 06:25 +0200, Denys Fedoryshchenko wrote:
> On 2015-11-04 00:06, Cong Wang wrote:
> > On Mon, Nov 2, 2015 at 6:11 AM, Denys Fedoryshchenko
> > <nuclearcat@nuclearcat.com> wrote:
> >> Hi!
> >> 
> >> Actually seems i was getting this panic for a while (once per week) on
> >> loaded pppoe server, but just now was able to get full panic message.
> >> After checking commit logs on sch_fq.c i didnt seen any fixes, so 
> >> probably
> >> upgrading to newer kernel wont help?
> > 
> > 
> > Can you share your `tc qdisc show dev xxxx` with us? And how to 
> > reproduce
> > it? I tried to setup htb+fq and then flip the interface back and forth
> > but I don't
> > see any crash.
> My guess it wont be easy to reproduce, it is happening on box with 4.5k 
> interfaces, that constantly create/delete interfaces,
> and even with that this problem may happen once per day, or may not 
> happen for 1 week.
> 
> Here is script that is being fired after new ppp interface detected. But 
> pppoe process are independent from
> process that are "establishing" shapers.


It is probably a generic bug. sch_fq seems OK to me.

Somehow nobody tries to change qdisc hundred times per second ;)

Could you try following patch ?

It seems to 'fix' the issue for me.

diff --git a/net/core/dev.c b/net/core/dev.c
index 8ce3f74cd6b9..bf136103bc7b 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2880,6 +2880,12 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 		spin_lock(&q->busylock);
 
 	spin_lock(root_lock);
+	if (unlikely(q != rcu_dereference_bh(txq->qdisc))) {
+		pr_err_ratelimited("Arg, qdisc changed ! state %lx\n", q->state);
+		kfree_skb(skb);
+		rc = NET_XMIT_DROP;
+		goto end;
+	}
 	if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
 		kfree_skb(skb);
 		rc = NET_XMIT_DROP;
@@ -2913,6 +2919,7 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 			__qdisc_run(q);
 		}
 	}
+end:
 	spin_unlock(root_lock);
 	if (unlikely(contended))
 		spin_unlock(&q->busylock);

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Eric Dumazet]

On Tue, 2015-11-03 at 20:46 -0800, Eric Dumazet wrote:
> On Wed, 2015-11-04 at 06:25 +0200, Denys Fedoryshchenko wrote:
> > On 2015-11-04 00:06, Cong Wang wrote:
> > > On Mon, Nov 2, 2015 at 6:11 AM, Denys Fedoryshchenko
> > > <nuclearcat@nuclearcat.com> wrote:
> > >> Hi!
> > >> 
> > >> Actually seems i was getting this panic for a while (once per week) on
> > >> loaded pppoe server, but just now was able to get full panic message.
> > >> After checking commit logs on sch_fq.c i didnt seen any fixes, so 
> > >> probably
> > >> upgrading to newer kernel wont help?
> > > 
> > > 
> > > Can you share your `tc qdisc show dev xxxx` with us? And how to 
> > > reproduce
> > > it? I tried to setup htb+fq and then flip the interface back and forth
> > > but I don't
> > > see any crash.
> > My guess it wont be easy to reproduce, it is happening on box with 4.5k 
> > interfaces, that constantly create/delete interfaces,
> > and even with that this problem may happen once per day, or may not 
> > happen for 1 week.
> > 
> > Here is script that is being fired after new ppp interface detected. But 
> > pppoe process are independent from
> > process that are "establishing" shapers.
> 
> 
> It is probably a generic bug. sch_fq seems OK to me.
> 
> Somehow nobody tries to change qdisc hundred times per second ;)
> 
> Could you try following patch ?
> 
> It seems to 'fix' the issue for me.

Following patch would be more appropriate.
Prior one was meant to 'show' the issue.

diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index cb5d4ad32946..7f5f3e8a10f5 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -706,9 +706,11 @@ struct Qdisc *dev_graft_qdisc(struct netdev_queue *dev_queue,
 	spin_lock_bh(root_lock);
 
 	/* Prune old scheduler */
-	if (oqdisc && atomic_read(&oqdisc->refcnt) <= 1)
-		qdisc_reset(oqdisc);
-
+	if (oqdisc) {
+		if (atomic_read(&oqdisc->refcnt) <= 1)
+			qdisc_reset(oqdisc);
+		set_bit(__QDISC_STATE_DEACTIVATED, &oqdisc->state);
+	}
 	/* ... and graft new one */
 	if (qdisc == NULL)
 		qdisc = &noop_qdisc;

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Denys Fedoryshchenko]

On 2015-11-04 06:58, Eric Dumazet wrote:
> On Tue, 2015-11-03 at 20:46 -0800, Eric Dumazet wrote:
>> On Wed, 2015-11-04 at 06:25 +0200, Denys Fedoryshchenko wrote:
>> > On 2015-11-04 00:06, Cong Wang wrote:
>> > > On Mon, Nov 2, 2015 at 6:11 AM, Denys Fedoryshchenko
>> > > <nuclearcat@nuclearcat.com> wrote:
>> > >> Hi!
>> > >>
>> > >> Actually seems i was getting this panic for a while (once per week) on
>> > >> loaded pppoe server, but just now was able to get full panic message.
>> > >> After checking commit logs on sch_fq.c i didnt seen any fixes, so
>> > >> probably
>> > >> upgrading to newer kernel wont help?
>> > >
>> > >
>> > > Can you share your `tc qdisc show dev xxxx` with us? And how to
>> > > reproduce
>> > > it? I tried to setup htb+fq and then flip the interface back and forth
>> > > but I don't
>> > > see any crash.
>> > My guess it wont be easy to reproduce, it is happening on box with 4.5k
>> > interfaces, that constantly create/delete interfaces,
>> > and even with that this problem may happen once per day, or may not
>> > happen for 1 week.
>> >
>> > Here is script that is being fired after new ppp interface detected. But
>> > pppoe process are independent from
>> > process that are "establishing" shapers.
>> 
>> 
>> It is probably a generic bug. sch_fq seems OK to me.
>> 
>> Somehow nobody tries to change qdisc hundred times per second ;)
>> 
>> Could you try following patch ?
>> 
>> It seems to 'fix' the issue for me.
> 
> Following patch would be more appropriate.
> Prior one was meant to 'show' the issue.
> 
> diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> index cb5d4ad32946..7f5f3e8a10f5 100644
> --- a/net/sched/sch_generic.c
> +++ b/net/sched/sch_generic.c
> @@ -706,9 +706,11 @@ struct Qdisc *dev_graft_qdisc(struct netdev_queue
> *dev_queue,
>  	spin_lock_bh(root_lock);
> 
>  	/* Prune old scheduler */
> -	if (oqdisc && atomic_read(&oqdisc->refcnt) <= 1)
> -		qdisc_reset(oqdisc);
> -
> +	if (oqdisc) {
> +		if (atomic_read(&oqdisc->refcnt) <= 1)
> +			qdisc_reset(oqdisc);
> +		set_bit(__QDISC_STATE_DEACTIVATED, &oqdisc->state);
> +	}
>  	/* ... and graft new one */
>  	if (qdisc == NULL)
>  		qdisc = &noop_qdisc;

Applied, will test it, but this bug might be triggered rarely.
I will try to push it to more pppoe servers in order to stress test them 
(and 4.3) more.

Re: kernel panic in 4.2.3, rb_erase in sch_fq

[Denys Fedoryshchenko]

I can confirm, after patch this issue never appeared again. So maybe 
good to push it to stable and etc :) Thanks a lot Eric, you saved me 
again.


Still i have some weird panic issues, maybe related to conntrack, but 
they are rare even on high load, so i am slowly gathering data, and i 
found at least one more person with similar conntrack crashes on latest 
kernels.


On 2015-11-04 06:46, Eric Dumazet wrote:
> On Wed, 2015-11-04 at 06:25 +0200, Denys Fedoryshchenko wrote:
>> On 2015-11-04 00:06, Cong Wang wrote:
>> > On Mon, Nov 2, 2015 at 6:11 AM, Denys Fedoryshchenko
>> > <nuclearcat@nuclearcat.com> wrote:
>> >> Hi!
>> >>
>> >> Actually seems i was getting this panic for a while (once per week) on
>> >> loaded pppoe server, but just now was able to get full panic message.
>> >> After checking commit logs on sch_fq.c i didnt seen any fixes, so
>> >> probably
>> >> upgrading to newer kernel wont help?
>> >
>> >
>> > Can you share your `tc qdisc show dev xxxx` with us? And how to
>> > reproduce
>> > it? I tried to setup htb+fq and then flip the interface back and forth
>> > but I don't
>> > see any crash.
>> My guess it wont be easy to reproduce, it is happening on box with 
>> 4.5k
>> interfaces, that constantly create/delete interfaces,
>> and even with that this problem may happen once per day, or may not
>> happen for 1 week.
>> 
>> Here is script that is being fired after new ppp interface detected. 
>> But
>> pppoe process are independent from
>> process that are "establishing" shapers.
> 
> 
> It is probably a generic bug. sch_fq seems OK to me.
> 
> Somehow nobody tries to change qdisc hundred times per second ;)
> 
> Could you try following patch ?
> 
> It seems to 'fix' the issue for me.
> 
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 8ce3f74cd6b9..bf136103bc7b 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -2880,6 +2880,12 @@ static inline int __dev_xmit_skb(struct sk_buff
> *skb, struct Qdisc *q,
>  		spin_lock(&q->busylock);
> 
>  	spin_lock(root_lock);
> +	if (unlikely(q != rcu_dereference_bh(txq->qdisc))) {
> +		pr_err_ratelimited("Arg, qdisc changed ! state %lx\n", q->state);
> +		kfree_skb(skb);
> +		rc = NET_XMIT_DROP;
> +		goto end;
> +	}
>  	if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
>  		kfree_skb(skb);
>  		rc = NET_XMIT_DROP;
> @@ -2913,6 +2919,7 @@ static inline int __dev_xmit_skb(struct sk_buff
> *skb, struct Qdisc *q,
>  			__qdisc_run(q);
>  		}
>  	}
> +end:
>  	spin_unlock(root_lock);
>  	if (unlikely(contended))
>  		spin_unlock(&q->busylock);