Re: 5.15-rc3+ crash in fq-codel?

[Ben Greear <greearb@candelatech.com>]

On 9/28/21 4:25 PM, Eric Dumazet wrote:
> 
> 
> On 9/28/21 3:00 PM, Ben Greear wrote:
>> On 9/27/21 5:16 PM, Ben Greear wrote:
>>> On 9/27/21 5:04 PM, Ben Greear wrote:
>>>> On 9/27/21 4:49 PM, Eric Dumazet wrote:
>>>>>
>>>>>
>>>>> On 9/27/21 4:30 PM, Ben Greear wrote:
>>>>>> Hello,
>>>>>>
>>>>>> In a hacked upon kernel, I'm getting crashes in fq-codel when doing bi-directional
>>>>>> pktgen traffic on top of mac-vlans.  Unfortunately for me, I've made big changes to
>>>>>> pktgen so I cannot easily run this test on stock kernels, and there is some chance
>>>>>> some of my hackings have caused this issue.
>>>>>>
>>>>>> But, in case others have seen similar, please let me know.  I shall go digging
>>>>>> in the meantime...
>>>>>>
>>>>>> Looks to me like 'skb' is NULL in line 120 below.
>>>>>
>>>>>
>>>>> pktgen must not be used in a mode where a single skb
>>>>> is cloned and reused, if packet needs to be stored in a qdisc.
>>>>>
>>>>> qdisc of all sorts assume skb->next/prev can be used as
>>>>> anchor in their list.
>>>>>
>>>>> If the same skb is queued multiple times, lists are corrupted.
>>>>>
>>>>> Please double check your clone_skb pktgen setup.
>>>>>
>>>>> I thought we had IFF_TX_SKB_SHARING for this, and that macvlan was properly clearing this bit.
>>>>
>>>> My pktgen config was not using any duplicated queueing in this case.
>>>>
>>>> I changed to pfifo fast and so far it is stable for ~10 minutes, where before it would crash
>>>> within a minute.  I'll let it bake overnight....
>>>
>>> Still running stable.  I also notice we have been using fq-codel for a while and haven't noticed
>>> this problem (next most recent kernel we might have run similar test on would be 5.13-ish).
>>>
>>> I'll duplicate this test on our older kernels tomorrow to see if it looks like a regression or
>>> if we just haven't actually done this exact test in a while...
>>
>> We can reproduce this crash as far back as 5.4 using fq-codel, with our pktgen driving mac-vlans.
>> We did not try any kernels older than 5.4.
>> We cannot reproduce with pfifo on 5.15-rc3 on an overnight run.
>> We cannot produce with user-space UDP traffic on any kernel/qdisc combination.
>> Our pktgen is configured for multi-skb of 0 (no multiple submits of the same skb)
>>
>> While looking briefly at fq-codel, I didn't notice any locking in the code that crashed.
>> Any chance that it makes assumptions that would be incorrect with pktgen running multiple
>> threads (one thread per mac-vlan) on top of a single qdisc belonging to the underlying NIC?
>>
> 
> 
> qdisc are protected by a qdisc spinlock.
> 
> fq-codel does not have to lock anything in its enqueue() and dequeue() methods.
> 
> I guess your local changes to pktgen might be to blame.
> 
> pfifo is much simpler than fq-codel, it uses less fields from skb.

I looked through my pktgen, and the skb creation and setup code looks pretty
similar to upstream pktgen.

I also added this debugging code:

[greearb@ben-dt4 linux-5.15.dev.y]$ git diff
diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c
index bb0cd6d3d2c2..56e22106e19d 100644
--- a/net/sched/sch_fq_codel.c
+++ b/net/sched/sch_fq_codel.c
@@ -165,6 +165,11 @@ static unsigned int fq_codel_drop(struct Qdisc *sch, unsigned int max_packets,
         len = 0;
         i = 0;
         do {
+               if (!flow->head) {
+                       pr_err("fq-codel-drop: idx: %d maxbacklog: %d  threshold: %d max_packets: %d len: %d i: %d\n",
+                              idx, maxbacklog, threshold, max_packets, len, i);
+                       BUG_ON(1);
+               }
                 skb = dequeue_head(flow);
                 len += qdisc_pkt_len(skb);
                 mem += get_codel_cb(skb)->mem_usage;

The printout I see when this hits is:


fq-codel-drop: idx: 955 maxbacklog: 7756222  threshold: 3878111 max_packets: 64 len: 93868 i: 62
kernel BUG at net/sched/sch_fq_codel.c:171!
.....

So, I guess this means that the backlog byte counter is out of sync with the packet queue somehow?

Any suggestions for what kinds of issues in pktgen could cause this?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com