From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C49BC433ED for ; Mon, 12 Apr 2021 23:09:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 18E6A61350 for ; Mon, 12 Apr 2021 23:09:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241965AbhDLXJ1 (ORCPT ); Mon, 12 Apr 2021 19:09:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237611AbhDLXJ1 (ORCPT ); Mon, 12 Apr 2021 19:09:27 -0400 Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 251EEC061574 for ; Mon, 12 Apr 2021 16:09:08 -0700 (PDT) Received: by mail-lj1-x22c.google.com with SMTP id a25so4019748ljm.11 for ; Mon, 12 Apr 2021 16:09:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=waldekranz-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=n38TXfinM8M/vOCnLljCyW6opWl632LLNTqZhEwUeTA=; b=R9NgDMrVFMIUl7QxV+c06S2YIy6SIecVB3jtjAgbpJZl9WjI9RKyMOFYx/6i5MPXsJ wYY5N/WjMEiro2ZpcpLzYRCd8sJ+xJqanf9Kn3N8U61/k4nRAtxj9heLVLpDhF2UMS/3 i1UXBwCVWZ+eUxFfJoOfVDNDNBeo6mC66kaeaHwa/xhraLqYsiHa4/aPxcC9muqxyTvV cscwohBmetNs0dBN+EnoBDcBTbntLaaingqU6I0rtOItqkHIuf4j1sB5v2ftJ15/2DGd l7h3aBMhTDeDrrYnnlM1IKHM58FGbmmwClM13s2mk1CkxoCmq74M4AUK3pXTmciSnGpU cs4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=n38TXfinM8M/vOCnLljCyW6opWl632LLNTqZhEwUeTA=; b=HRXcM0k14bLIKlKEzziLT2QHbYhzjTKa2fc0Pe1fcqr7J6UE3vWiu/CYD4O7+0InOW dTwAZ/MOthLjByqbEgLDtv5baX9hPO2+kLYh0gcF5sQWJK3X6V9j8HGvywSPrQ79E7xw iktuaDof2RsDt+kISuAFtoafQhaBRrQY2eHWb4kiIMg5RS4XPNDJgZU0JrtzU0iWieQl elVg0LpdkYIhcW5MpoUqRUOinMlmeTFvMZTUx0U6sIDJ80rYRoKWLsbcHLZr3jD/4M4r CRCXJVXofQ9zOjWUkZMmiLQKQCm+FqeS0NtmaIA0VjB6KbF6+zsCNH6rCVbfoFh5m9Fj KSvw== X-Gm-Message-State: AOAM53194h5sEAxEm/g1K+uOh0sVlpI1mySK6+kFocArIDWnAoHAzZUp /JOd8b8LQAp6698hI2BN5CKDJw== X-Google-Smtp-Source: ABdhPJx06NxMuV1QKmqrbCvLvPEzOj/x6kSDnfOLDli9t9gNtmoveeX6VknvaWTDksfG253aQSfgZA== X-Received: by 2002:a2e:99c1:: with SMTP id l1mr707785ljj.180.1618268946482; Mon, 12 Apr 2021 16:09:06 -0700 (PDT) Received: from wkz-x280 (h-90-88.A259.priv.bahnhof.se. [212.85.90.88]) by smtp.gmail.com with ESMTPSA id m15sm2652921lfg.43.2021.04.12.16.09.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Apr 2021 16:09:06 -0700 (PDT) From: Tobias Waldekranz To: Marek Behun Cc: Vladimir Oltean , Ansuel Smith , netdev@vger.kernel.org, "David S. Miller" , Jakub Kicinski , Andrew Lunn , Vivien Didelot , Florian Fainelli , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eric Dumazet , Wei Wang , Cong Wang , Taehee Yoo , =?utf-8?B?QmrDtnJuIFTDtnBlbA==?= , zhang kai , Weilong Chen , Roopa Prabhu , Di Zhu , Francis Laniel , linux-kernel@vger.kernel.org Subject: Re: [PATCH RFC net-next 0/3] Multi-CPU DSA support In-Reply-To: <20210413005518.2f9b9cef@thinkpad> References: <20210410133454.4768-1-ansuelsmth@gmail.com> <20210411200135.35fb5985@thinkpad> <20210411185017.3xf7kxzzq2vefpwu@skbuf> <878s5nllgs.fsf@waldekranz.com> <20210412213045.4277a598@thinkpad> <8735vvkxju.fsf@waldekranz.com> <20210412235054.73754df9@thinkpad> <87wnt7jgzk.fsf@waldekranz.com> <20210413005518.2f9b9cef@thinkpad> Date: Tue, 13 Apr 2021 01:09:05 +0200 Message-ID: <87r1jfje26.fsf@waldekranz.com> MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Tue, Apr 13, 2021 at 00:55, Marek Behun wrote: > On Tue, 13 Apr 2021 00:05:51 +0200 > Tobias Waldekranz wrote: > >> On Mon, Apr 12, 2021 at 23:50, Marek Behun wrote: >> > On Mon, 12 Apr 2021 23:22:45 +0200 >> > Tobias Waldekranz wrote: >> > >> >> On Mon, Apr 12, 2021 at 21:30, Marek Behun wrote: >> >> > On Mon, 12 Apr 2021 14:46:11 +0200 >> >> > Tobias Waldekranz wrote: >> >> > >> >> >> I agree. Unless you only have a few really wideband flows, a LAG will >> >> >> typically do a great job with balancing. This will happen without the >> >> >> user having to do any configuration at all. It would also perform well >> >> >> in "router-on-a-stick"-setups where the incoming and outgoing port is >> >> >> the same. >> >> > >> >> > TLDR: The problem with LAGs how they are currently implemented is that >> >> > for Turris Omnia, basically in 1/16 of configurations the traffic would >> >> > go via one CPU port anyway. >> >> > >> >> > >> >> > >> >> > One potencial problem that I see with using LAGs for aggregating CPU >> >> > ports on mv88e6xxx is how these switches determine the port for a >> >> > packet: only the src and dst MAC address is used for the hash that >> >> > chooses the port. >> >> > >> >> > The most common scenario for Turris Omnia, for example, where we have 2 >> >> > CPU ports and 5 user ports, is that into these 5 user ports the user >> >> > plugs 5 simple devices (no switches, so only one peer MAC address for >> >> > port). So we have only 5 pairs of src + dst MAC addresses. If we simply >> >> > fill the LAG table as it is done now, then there is 2 * 0.5^5 = 1/16 >> >> > chance that all packets would go through one CPU port. >> >> > >> >> > In order to have real load balancing in this scenario, we would either >> >> > have to recompute the LAG mask table depending on the MAC addresses, or >> >> > rewrite the LAG mask table somewhat randomly periodically. (This could >> >> > be in theory offloaded onto the Z80 internal CPU for some of the >> >> > switches of the mv88e6xxx family, but not for Omnia.) >> >> >> >> I thought that the option to associate each port netdev with a DSA >> >> master would only be used on transmit. Are you saying that there is a >> >> way to configure an mv88e6xxx chip to steer packets to different CPU >> >> ports depending on the incoming port? >> >> >> >> The reason that the traffic is directed towards the CPU is that some >> >> kind of entry in the ATU says so, and the destination of that entry will >> >> either be a port vector or a LAG. Of those two, only the LAG will offer >> >> any kind of balancing. What am I missing? >> > >> > Via port vectors you can "load balance" by ports only, i.e. input port X >> > -> trasmit via CPU port Y. >> >> How is this done? In a case where there is no bridging between the >> ports, then I understand. Each port could have its own FID. But if you >> have this setup... >> >> br0 wan >> / \ >> lan0 lan1 >> >> lan0 and lan1 would use the same FID. So how could you say that frames >> from lan0 should go to cpu0 and frames from lan1 should go to cpu1 if >> the DA is the same? What would be the content of the ATU in a setup >> like that? >> >> > When using LAGs, you are load balancing via hash(src MAC | dst mac) >> > only. This is better in some ways. But what I am saying is that if the >> > LAG mask table is static, as it is now implemented in mv88e6xxx code, >> > then for many scenarios there is a big probability of no load balancing >> > at all. For Turris Omnia for example there is 6.25% probability that >> > the switch chip will send all traffic to the CPU via one CPU port. >> > This is because the switch chooses the LAG port only from the hash of >> > dst+src MAC address. (By the 1/16 = 6.25% probability I mean that for >> > cca 1 in 16 customers, the switch would only use one port when sending >> > data to the CPU). >> > >> > The round robin solution here is therefore better in this case. >> >> I agree that it would be better in that case. I just do not get how you >> get the switch to do it for you. > > I thought that this is configured in the mv88e6xxx_port_vlan() function. > For each port, you specify via which ports data can egress. > So for ports 0, 2, 4 you can enable CPU port 0, and for ports 1 and 3 > CPU port 1. Ahh I see. Well the port based VLANs are just an isolation mechanism. So with a setup like this... cpu0 cpu1 lan0 lan1 cpu0 x cpu1 x lan0 x x lan1 x x ...you could get the isolation in place. But you will still lookup the DA in the ATU, and there you will find a destination of either cpu0 or cpu1. So for one of the ports, the destination will be outside of its port based VLAN. Once the vectors are ANDed together, it is left with no valid port to egress through, and the packet is dropped. > Am I wrong? I confess that I did not understand this into the most fine > details, so it is entirely possible that I am missing something > important and am completely wrong. Maybe this cannot be done. I really doubt that it can be done. Not in any robust way at least. Happy to be proven wrong though! :)