From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5396C433F5 for ; Tue, 28 Sep 2021 00:04:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 879D4611CE for ; Tue, 28 Sep 2021 00:04:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238268AbhI1AFo (ORCPT ); Mon, 27 Sep 2021 20:05:44 -0400 Received: from dispatch1-us1.ppe-hosted.com ([148.163.129.52]:42538 "EHLO dispatch1-us1.ppe-hosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238236AbhI1AFn (ORCPT ); Mon, 27 Sep 2021 20:05:43 -0400 X-Virus-Scanned: Proofpoint Essentials engine Received: from mx1-us1.ppe-hosted.com (unknown [10.7.66.132]) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id CE7582005E; Tue, 28 Sep 2021 00:04:03 +0000 (UTC) Received: from mail3.candelatech.com (mail2.candelatech.com [208.74.158.173]) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTP id 9841C94006A; Tue, 28 Sep 2021 00:04:03 +0000 (UTC) Received: from [192.168.100.195] (50-251-239-81-static.hfc.comcastbusiness.net [50.251.239.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail3.candelatech.com (Postfix) with ESMTPSA id 165CF13C2B0; Mon, 27 Sep 2021 17:04:02 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 mail3.candelatech.com 165CF13C2B0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=candelatech.com; s=default; t=1632787443; bh=yoyxY9NQdYCyiThx6d7oTM2+W3W2iyR/7a/1CZ68Ab8=; h=Subject:To:References:From:Date:In-Reply-To:From; b=gT7zAq+xDKyiaLtz+bUUT2eRfxGcaTRBFCJxC6gMrOm3L1slk+SYbIVZaSfXr4LXs S3jXLreqMIFR27MX0zMY+d1B4bz0KIQUQ5Cs/D5B44DHKNpknEEEj5OcD5nRGJqHQw CuYG2rDoowPsHbZ1XRiEJGccQ/IfkTWYcv5cYzvo= Subject: Re: 5.15-rc3+ crash in fq-codel? To: Eric Dumazet , netdev References: From: Ben Greear Organization: Candela Technologies Message-ID: <00e495ba-391e-6ad8-94a2-930fbc826a37@candelatech.com> Date: Mon, 27 Sep 2021 17:04:02 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-MDID: 1632787444-4olwz901Xspa Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 9/27/21 4:49 PM, Eric Dumazet wrote: > > > On 9/27/21 4:30 PM, Ben Greear wrote: >> Hello, >> >> In a hacked upon kernel, I'm getting crashes in fq-codel when doing bi-directional >> pktgen traffic on top of mac-vlans.  Unfortunately for me, I've made big changes to >> pktgen so I cannot easily run this test on stock kernels, and there is some chance >> some of my hackings have caused this issue. >> >> But, in case others have seen similar, please let me know.  I shall go digging >> in the meantime... >> >> Looks to me like 'skb' is NULL in line 120 below. > > > pktgen must not be used in a mode where a single skb > is cloned and reused, if packet needs to be stored in a qdisc. > > qdisc of all sorts assume skb->next/prev can be used as > anchor in their list. > > If the same skb is queued multiple times, lists are corrupted. > > Please double check your clone_skb pktgen setup. > > I thought we had IFF_TX_SKB_SHARING for this, and that macvlan was properly clearing this bit. My pktgen config was not using any duplicated queueing in this case. I changed to pfifo fast and so far it is stable for ~10 minutes, where before it would crash within a minute. I'll let it bake overnight.... Thanks, Ben > >> >> For help, type "help". >> Type "apropos word" to search for commands related to "word"... >> Reading symbols from ./net/sched/sch_fq_codel.ko...done. >> "/home/greearb/kernel/2.6/linux-5.15.x64/vmlinux" is not a core dump: file format not recognized >> (gdb) l *(fq_codel_enqueue+0x24b) >> 0x76b is in fq_codel_enqueue (/home/greearb/git/linux-5.15.dev.y/net/sched/sch_fq_codel.c:120). >> 115    /* remove one skb from head of slot queue */ >> 116    static inline struct sk_buff *dequeue_head(struct fq_codel_flow *flow) >> 117    { >> 118        struct sk_buff *skb = flow->head; >> 119 >> 120        flow->head = skb->next; >> 121        skb_mark_not_on_list(skb); >> 122        return skb; >> 123    } >> 124 >> (gdb) >> >> >> BUG: kernel NULL pointer dereference, address: 0000000000000000 >> #PF: supervisor read access in kernel mode >> #PF: error_code(0x0000) - not-present page >> PGD 0 P4D 0 >> Oops: 0000 [#1] PREEMPT SMP >> CPU: 3 PID: 2077 Comm: kpktgend_3 Not tainted 5.15.0-rc3+ #2 >> Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/19/2019 >> RIP: 0010:fq_codel_enqueue+0x24b/0x380 [sch_fq_codel] >> Code: e0 02 48 89 44 24 08 49 c1 e0 06 4c 03 83 50 01 00 00 45 31 f6 45 31 c9 31 c9 89 74 24 10 eb 04 39 fa 73 33 49 8b 00 83 c13 >> RSP: 0018:ffffc9000030fd10 EFLAGS: 00010202 >> RAX: 0000000000000000 RBX: ffff88810a78f600 RCX: 0000000000000032 >> RDX: 00000000000121ca RSI: ffff88812d716900 RDI: 00000000003b26f5 >> RBP: ffffc9000030fd78 R08: ffff8881311dd340 R09: 00000000000121ca >> R10: 000000000000034d R11: 0000000001680900 R12: ffffc9000030fde0 >> R13: 000000000001b900 R14: 000000000001b900 R15: 0000000000000040 >> FS:  0000000000000000(0000) GS:ffff888265cc0000(0000) knlGS:0000000000000000 >> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 0000000000000000 CR3: 000000000260f003 CR4: 00000000003706e0 >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> Call Trace: >>  dev_qdisc_enqueue+0x35/0x90 >>  __dev_queue_xmit+0x647/0xb70 >>  macvlan_start_xmit+0x4a/0x110 [macvlan] >>  pktgen_thread_worker+0x19fe/0x20ed [pktgen] >>  ? wait_woken+0x60/0x60 >>  ? pktgen_rem_all_ifs+0x70/0x70 [pktgen] >>  kthread+0x11e/0x150 >>  ? set_kthread_struct+0x40/0x40 >>  ret_from_fork+0x1f/0x30 >> >> >> Thanks, >> Ben >> > -- Ben Greear Candela Technologies Inc http://www.candelatech.com