From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 684BAC43461 for ; Fri, 19 Mar 2021 18:16:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A8FD6195C for ; Fri, 19 Mar 2021 18:16:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230341AbhCSSP5 (ORCPT ); Fri, 19 Mar 2021 14:15:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229956AbhCSSP0 (ORCPT ); Fri, 19 Mar 2021 14:15:26 -0400 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6800C06175F; Fri, 19 Mar 2021 11:15:25 -0700 (PDT) Received: by mail-pg1-x532.google.com with SMTP id k24so4231329pgl.6; Fri, 19 Mar 2021 11:15:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=yApSLu496m7gvqSRhwz6HeRoNaU50Bo27wlXzUIZtfg=; b=eehFhYe7ZsCKZWptxyH0GSZhy5Czr+VIm5/5FElKHxrK/4P3sYqetHdmkKeorMqAYi SOqlnw7HVehgKaV9C3Kyg2dqeOXMpmUGRwPDbPWy8Wj60BO5RYzhJuDbY6apBrJIoKkJ vsDoSq9U5qjx7mrQtiO7xxMr3a7sPCgcEsRBofgPCNCKu4+fl90GquyGAIelLzNL2/tu o2D3UCZjwpL9gZwPwvnYNWadT/m0WXeJH7wAl/giSchVh6Zuk0qB0t++wjck5fDXCmjq K8hCc5k+e9VqSZKvXaaK7CfRxCJFfhTb3Djc0TjK13IhEknLZhDgVaBAnXLUV0lC2o3n W7bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=yApSLu496m7gvqSRhwz6HeRoNaU50Bo27wlXzUIZtfg=; b=reOf36TGQ7b21vv9PZWNE+e3qJwoNYfeH3iVPfHV7zTncH1cXadKdWUjjV7A0pNwn6 eEiml+4gfqsN10wQgiWEqpOtpsgjbak+jIwadP3Lwv5k46lNF84Uh8GkHT1lzvrhuv5W Y35uPmu3PaslpvMxmsc9Vc8tG+dKaPCG2JLFlAcoJrm8Xzp87pZP8IPs/lk1Sk7ei/29 J7qT0G9EMp7kB0Nx2ohWKMGz2WuvL0vaBbV9FOFtgyynhC5wAvQrJJ/A3rbM0RM2inP2 tDJKEgg7mPvC+rKzbnXZE20hwrYvlT5AEQ9NUdNbJUm7WEr38CYzfVwEirM7UfKyVGDh xUww== X-Gm-Message-State: AOAM533iZQoOv9U7UXmJsGwYGv5sCSSEZecGnfVNQmxX/lIfzXK6XXTB qMT3IGHUJ1c4AyIi15G0n6Fuj8Jzp6bgr8HIAMs= X-Google-Smtp-Source: ABdhPJynWk83Rk87ejofNY38HDvwGkAlAwC/1NnuFdTn/Q4bqR54h7dRTJcA85qzFKeZsG/59qaCjB39fsY4RX26G+Q= X-Received: by 2002:aa7:85c1:0:b029:1f4:4fcc:384d with SMTP id z1-20020aa785c10000b02901f44fcc384dmr10609791pfn.10.1616177725389; Fri, 19 Mar 2021 11:15:25 -0700 (PDT) MIME-Version: 1.0 References: <1615603667-22568-1-git-send-email-linyunsheng@huawei.com> <1615777818-13969-1-git-send-email-linyunsheng@huawei.com> <20210315115332.1647e92b@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> <87eegddhsj.fsf@toke.dk> <3bae7b26-9d7f-15b8-d466-ff5c26d08b35@huawei.com> In-Reply-To: <3bae7b26-9d7f-15b8-d466-ff5c26d08b35@huawei.com> From: Cong Wang Date: Fri, 19 Mar 2021 11:15:14 -0700 Message-ID: Subject: Re: [Linuxarm] Re: [RFC v2] net: sched: implement TCQ_F_CAN_BYPASS for lockless qdisc To: Yunsheng Lin Cc: "Jason A. Donenfeld" , =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= , Jakub Kicinski , David Miller , Vladimir Oltean , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eric Dumazet , Wei Wang , "Cong Wang ." , Taehee Yoo , Linux Kernel Network Developers , LKML , linuxarm@openeuler.org, Marc Kleine-Budde , linux-can@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Thu, Mar 18, 2021 at 12:33 AM Yunsheng Lin wrot= e: > > On 2021/3/17 21:45, Jason A. Donenfeld wrote: > > On 3/17/21, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >> Cong Wang writes: > >> > >>> On Mon, Mar 15, 2021 at 2:07 PM Jakub Kicinski wrot= e: > >>>> > >>>> I thought pfifo was supposed to be "lockless" and this change > >>>> re-introduces a lock between producer and consumer, no? > >>> > >>> It has never been truly lockless, it uses two spinlocks in the ring > >>> buffer > >>> implementation, and it introduced a q->seqlock recently, with this pa= tch > >>> now we have priv->lock, 4 locks in total. So our "lockless" qdisc end= s > >>> up having more locks than others. ;) I don't think we are going to a > >>> right direction... > >> > >> Just a thought, have you guys considered adopting the lockless MSPC ri= ng > >> buffer recently introduced into Wireguard in commit: > >> > >> 8b5553ace83c ("wireguard: queueing: get rid of per-peer ring buffers") > >> > >> Jason indicated he was willing to work on generalising it into a > >> reusable library if there was a use case for it. I haven't quite thoug= h > >> through the details of whether this would be such a use case, but > >> figured I'd at least mention it :) > > > > That offer definitely still stands. Generalization sounds like a lot of= fun. > > > > Keep in mind though that it's an eventually consistent queue, not an > > immediately consistent one, so that might not match all use cases. It > > works with wg because we always trigger the reader thread anew when it > > finishes, but that doesn't apply to everyone's queueing setup. > > Thanks for mentioning this. > > "multi-producer, single-consumer" seems to match the lockless qdisc's > paradigm too, for now concurrent enqueuing/dequeuing to the pfifo_fast's > queues() is not allowed, it is protected by producer_lock or consumer_loc= k. > > So it would be good to has lockless concurrent enqueuing, while dequeuing > can be protected by qdisc_lock() or q->seqlock, which meets the "multi-pr= oducer, > single-consumer" paradigm. I don't think so. Usually we have one queue for each CPU so we can expect each CPU has a lockless qdisc assigned, but we can not assume this in the code, so we still have to deal with multiple CPU's sharing a lockless q= disc, and we usually enqueue and dequeue in process context, so it means we could have multiple producers and multiple consumers. On the other hand, I don't think the problems we have been fixing are the r= ing buffer implementation itself, they are about the high-level qdisc state transitions. Thanks.