From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF639C43381 for ; Tue, 26 Feb 2019 23:19:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8C75A218DE for ; Tue, 26 Feb 2019 23:19:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hA/XHU2U" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729277AbfBZXTb (ORCPT ); Tue, 26 Feb 2019 18:19:31 -0500 Received: from mail-pl1-f179.google.com ([209.85.214.179]:43996 "EHLO mail-pl1-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728791AbfBZXTb (ORCPT ); Tue, 26 Feb 2019 18:19:31 -0500 Received: by mail-pl1-f179.google.com with SMTP id m10so6966298plt.10 for ; Tue, 26 Feb 2019 15:19:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=J3tpbs5Fkg2tc+NfAubshRgeqqcpuoLzEUTRi1W5KWo=; b=hA/XHU2U/6m9x6XS0tr6hXVi/Cf65IYsuNlH/5B3akH6w2iwXWMnK1Svzc4wBs0Gps nUoYC9+s6S7wvTd6AZCZt5plodZGw1HQYbByyfSpkJZiPPloOm+Z0kj7rnCKEuSfduBW BzZ+nMMmQrlt16c+IYug74QFiKSoQFKceWkNAnx3Cu/sDzvL095sYbaTRVColQAySAYN mhCH3edYHmw62rrst+VUjQH4U6PrviRtgR0gkROspFk5yfcvZ9nVvvfwNuelki727kX3 MI8sykDry2Z4lHGVbktWvcykypqPiXxn/Vkxzzb8ZSjQs60IH7mvJ4H9/mxfF1oxjRMJ 3y3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=J3tpbs5Fkg2tc+NfAubshRgeqqcpuoLzEUTRi1W5KWo=; b=XARF38yZ7TQDPKltXZCWTrvLDazAaDpYIcltxIaQeF1WYWSEDJKHGpOA4rcFe/H6lq 3qeTAqaTe4JNj17maAMFX9Hp3zhZAtUwE8m2RivueRtzQuFj5FGstfhLutZwT8y4eLZG u2eSsXYeVxj0/FRFLaXqcih1yLXwPowB3HUJMSMdgzCGCzSt0cwgqU0kBGqTDLGWj4BJ /Dqabumpvnno+T4tstgjtS9txTXbTE+LLvspZP+socpiBKY/yITQfFS/OwN24PmYd2uQ QWV8kUBWxYIS4Rw/DvzEDlwOM/JrtaHbiiI7e5JL8qUIPKudXbNz0z89b7cQd/U1qWjZ SI8A== X-Gm-Message-State: AHQUAuaUGLZQkN6TGBJPNO6j2MWlujqDaug470W5UIVmie9dXpkzjm2f +GSGA5wwzqUyEQZyV245Ckox0iIB X-Google-Smtp-Source: AHgI3IY4MtTHQs5sUMbK/PxN5k0PG6N3njB1pB7zQousVzCMDJ+WXGfDSPelKjLhtmWsyJvjmJm78g== X-Received: by 2002:a17:902:6508:: with SMTP id b8mr29257879plk.17.1551223170361; Tue, 26 Feb 2019 15:19:30 -0800 (PST) Received: from ?IPv6:2620:15c:2c1:200:55c7:81e6:c7d8:94b? ([2620:15c:2c1:200:55c7:81e6:c7d8:94b]) by smtp.gmail.com with ESMTPSA id 63sm30691019pfs.65.2019.02.26.15.19.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Feb 2019 15:19:28 -0800 (PST) Subject: Re: [BUG] net/sched : qlen can not really be per cpu ? To: Eric Dumazet , John Fastabend , Networking , Jamal Hadi Salim , Cong Wang , Jiri Pirko References: <5fb87db3-d5bd-714b-213d-6821f4bf37da@gmail.com> From: Eric Dumazet Message-ID: Date: Tue, 26 Feb 2019 15:19:27 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <5fb87db3-d5bd-714b-213d-6821f4bf37da@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 02/25/2019 10:42 PM, Eric Dumazet wrote: > HTB + pfifo_fast as a leaf qdisc hits badly the following warning in htb_activate() : > > WARN_ON(cl->level || !cl->leaf.q || !cl->leaf.q->q.qlen); > > This is because pfifo_fast does not update sch->q.qlen, but per cpu counters. > So cl->leaf.q->q.qlen is zero. > > HFSC, CBQ, DRR, QFQ have the same problem. > > Any ideas how we can fix this ? What about something simple for stable ? ( I yet have to boot/test this ) diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index 9481f2c142e26ee1174653d673e6134edd9851da..3a9e442fcaaf2ea48ae65bc87ee95f59cd7100c8 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -51,7 +51,10 @@ struct qdisc_size_table { struct qdisc_skb_head { struct sk_buff *head; struct sk_buff *tail; - __u32 qlen; + union { + __u32 qlen; + atomic_t atomic_qlen; + }; spinlock_t lock; }; @@ -408,27 +411,19 @@ static inline void qdisc_cb_private_validate(const struct sk_buff *skb, int sz) BUILD_BUG_ON(sizeof(qcb->data) < sz); } -static inline int qdisc_qlen_cpu(const struct Qdisc *q) -{ - return this_cpu_ptr(q->cpu_qstats)->qlen; -} - static inline int qdisc_qlen(const struct Qdisc *q) { return q->q.qlen; } -static inline int qdisc_qlen_sum(const struct Qdisc *q) +static inline u32 qdisc_qlen_sum(const struct Qdisc *q) { - __u32 qlen = q->qstats.qlen; - int i; + u32 qlen = q->qstats.qlen; - if (q->flags & TCQ_F_NOLOCK) { - for_each_possible_cpu(i) - qlen += per_cpu_ptr(q->cpu_qstats, i)->qlen; - } else { + if (q->flags & TCQ_F_NOLOCK) + qlen += atomic_read(&q->q.atomic_qlen); + else qlen += q->q.qlen; - } return qlen; } @@ -827,12 +822,12 @@ static inline void qdisc_qstats_cpu_backlog_inc(struct Qdisc *sch, static inline void qdisc_qstats_cpu_qlen_inc(struct Qdisc *sch) { - this_cpu_inc(sch->cpu_qstats->qlen); + atomic_inc(&sch->q.atomic_qlen); } static inline void qdisc_qstats_cpu_qlen_dec(struct Qdisc *sch) { - this_cpu_dec(sch->cpu_qstats->qlen); + atomic_dec(&sch->q.atomic_qlen); } static inline void qdisc_qstats_cpu_requeues_inc(struct Qdisc *sch)