Re: 5.15-rc3+ crash in fq-codel?

From: Eric Dumazet <eric.dumazet@gmail.com>
To: Ben Greear <greearb@candelatech.com>, netdev <netdev@vger.kernel.org>
Subject: Re: 5.15-rc3+ crash in fq-codel?
Date: Mon, 27 Sep 2021 16:49:30 -0700	[thread overview]
Message-ID: <b6e8155e-7fae-16b0-59f0-2a2e6f5142de@gmail.com> (raw)
In-Reply-To: <dfa032f3-18f2-22a3-80bf-f0f570892478@candelatech.com>

On 9/27/21 4:30 PM, Ben Greear wrote:
> Hello,
> 
> In a hacked upon kernel, I'm getting crashes in fq-codel when doing bi-directional
> pktgen traffic on top of mac-vlans.  Unfortunately for me, I've made big changes to
> pktgen so I cannot easily run this test on stock kernels, and there is some chance
> some of my hackings have caused this issue.
> 
> But, in case others have seen similar, please let me know.  I shall go digging
> in the meantime...
> 
> Looks to me like 'skb' is NULL in line 120 below.

pktgen must not be used in a mode where a single skb
is cloned and reused, if packet needs to be stored in a qdisc.

qdisc of all sorts assume skb->next/prev can be used as
anchor in their list.

If the same skb is queued multiple times, lists are corrupted.

Please double check your clone_skb pktgen setup.

I thought we had IFF_TX_SKB_SHARING for this, and that macvlan was properly clearing this bit.

> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./net/sched/sch_fq_codel.ko...done.
> "/home/greearb/kernel/2.6/linux-5.15.x64/vmlinux" is not a core dump: file format not recognized
> (gdb) l *(fq_codel_enqueue+0x24b)
> 0x76b is in fq_codel_enqueue (/home/greearb/git/linux-5.15.dev.y/net/sched/sch_fq_codel.c:120).
> 115    /* remove one skb from head of slot queue */
> 116    static inline struct sk_buff *dequeue_head(struct fq_codel_flow *flow)
> 117    {
> 118        struct sk_buff *skb = flow->head;
> 119   
> 120        flow->head = skb->next;
> 121        skb_mark_not_on_list(skb);
> 122        return skb;
> 123    }
> 124   
> (gdb)
> 
> 
> BUG: kernel NULL pointer dereference, address: 0000000000000000
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP
> CPU: 3 PID: 2077 Comm: kpktgend_3 Not tainted 5.15.0-rc3+ #2
> Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/19/2019
> RIP: 0010:fq_codel_enqueue+0x24b/0x380 [sch_fq_codel]
> Code: e0 02 48 89 44 24 08 49 c1 e0 06 4c 03 83 50 01 00 00 45 31 f6 45 31 c9 31 c9 89 74 24 10 eb 04 39 fa 73 33 49 8b 00 83 c13
> RSP: 0018:ffffc9000030fd10 EFLAGS: 00010202
> RAX: 0000000000000000 RBX: ffff88810a78f600 RCX: 0000000000000032
> RDX: 00000000000121ca RSI: ffff88812d716900 RDI: 00000000003b26f5
> RBP: ffffc9000030fd78 R08: ffff8881311dd340 R09: 00000000000121ca
> R10: 000000000000034d R11: 0000000001680900 R12: ffffc9000030fde0
> R13: 000000000001b900 R14: 000000000001b900 R15: 0000000000000040
> FS:  0000000000000000(0000) GS:ffff888265cc0000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 000000000260f003 CR4: 00000000003706e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  dev_qdisc_enqueue+0x35/0x90
>  __dev_queue_xmit+0x647/0xb70
>  macvlan_start_xmit+0x4a/0x110 [macvlan]
>  pktgen_thread_worker+0x19fe/0x20ed [pktgen]
>  ? wait_woken+0x60/0x60
>  ? pktgen_rem_all_ifs+0x70/0x70 [pktgen]
>  kthread+0x11e/0x150
>  ? set_kthread_struct+0x40/0x40
>  ret_from_fork+0x1f/0x30
> 
> 
> Thanks,
> Ben
> 