From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71393C433E7 for ; Sat, 17 Oct 2020 05:54:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2E63D20709 for ; Sat, 17 Oct 2020 05:54:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Qz53maOY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436735AbgJQFyn (ORCPT ); Sat, 17 Oct 2020 01:54:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2411824AbgJQFym (ORCPT ); Sat, 17 Oct 2020 01:54:42 -0400 Received: from mail-il1-x142.google.com (mail-il1-x142.google.com [IPv6:2607:f8b0:4864:20::142]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC27FC05BD21 for ; Fri, 16 Oct 2020 21:37:40 -0700 (PDT) Received: by mail-il1-x142.google.com with SMTP id w17so4958230ilg.8 for ; Fri, 16 Oct 2020 21:37:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=iiGwrV0NKK1gY2dyfZjkPU9prcwn/lHJEnsM17MGAy0=; b=Qz53maOYpBBHWC/D2yPufBKX2d0XNPtUz7tXjDDyGbvr9i11nPR55My0VB4/9yRWOC adcnrm9PQfww+7npFVCIJNG+vTfsIWT5c5exiHH1EYJrNC9wIFmu6AmC3C09ZO9qUyIF f5823WFgbbqcxnpyaN6N7MPnknkLunP/UwzW918KvDEAQ8s2CaxomFIGmoGjTOF0rEQ/ VrZi8xCeZW261iB0ZlSth+EIcXuVeMC+TabfN8FOYGi56eHBI1bJlrEvZH71SPVSO+Ai RUgpNvNQ9t5+gupR7FKCyu3IutW5euMEIBTVVN5zE2WYsUGwFtliHnbnwjl5wggg2vu5 pZTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=iiGwrV0NKK1gY2dyfZjkPU9prcwn/lHJEnsM17MGAy0=; b=ppoxz76DImVXYDJ2i0QI92EnRiivCFa5zdo6+igEgAUZ7ymPz8oZCs6L+ss0JoD41y OUSLwkog+3qhsayKQbxzopJJ7lZGkMVd7KGBnsswZHjeNCoAJ5AYmXp+TLWiidXniGaI BfwRyknBHybMIHwG9xy0/cA/jKGMxYhsuG4uG9X7QXCxUvm0aORT7flwa6IUwy+Ap2xX HztMoNx1mBt5CRj9ilwbs0iEHVTrogFGaA3XXsa2D2bqs1AtQJ5MxalJcCrSqEmJWMrw RWpwI3Xz9ETWs5MMOd+TVNx6M+WUGzu7gl1G03AlJp3xPgKhqDtM0qwJP4kkoRbptyO1 4LaQ== X-Gm-Message-State: AOAM532PSjuj2Erdk2EJMK77k1ZqRUMRh9s38vHUHGHjpAGhOFpwfu0B j9LshfA+naCEE7kXjdjuvtJMHnZ3bIw= X-Google-Smtp-Source: ABdhPJzHvKd1Z61v12kZVQ2mwiAJMuIbqTBOze8k9RAEZugO53FTmmiIVEwq+Kg+gkRuDU3CpcEygg== X-Received: by 2002:a92:8742:: with SMTP id d2mr4945055ilm.153.1602909460270; Fri, 16 Oct 2020 21:37:40 -0700 (PDT) Received: from Davids-MacBook-Pro.local ([2601:282:803:7700:c015:614b:5c97:a602]) by smtp.googlemail.com with ESMTPSA id y75sm4274473iof.36.2020.10.16.21.37.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 16 Oct 2020 21:37:39 -0700 (PDT) Subject: Re: [PATCH net] nexthop: Fix performance regression in nexthop deletion To: Ido Schimmel , netdev@vger.kernel.org Cc: davem@davemloft.net, kuba@kernel.org, nikolay@nvidia.com, mlxsw@nvidia.com, Ido Schimmel References: <20201016172914.643282-1-idosch@idosch.org> From: David Ahern Message-ID: Date: Fri, 16 Oct 2020 22:37:38 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: <20201016172914.643282-1-idosch@idosch.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 10/16/20 11:29 AM, Ido Schimmel wrote: > From: Ido Schimmel > > While insertion of 16k nexthops all using the same netdev ('dummy10') > takes less than a second, deletion takes about 130 seconds: > > # time -p ip -b nexthop.batch > real 0.29 > user 0.01 > sys 0.15 > > # time -p ip link set dev dummy10 down > real 131.03 > user 0.06 > sys 0.52 > > This is because of repeated calls to synchronize_rcu() whenever a > nexthop is removed from a nexthop group: > > # /usr/share/bcc/tools/offcputime -p `pgrep -nx ip` -K > ... > b'finish_task_switch' > b'schedule' > b'schedule_timeout' > b'wait_for_completion' > b'__wait_rcu_gp' > b'synchronize_rcu.part.0' > b'synchronize_rcu' > b'__remove_nexthop' > b'remove_nexthop' > b'nexthop_flush_dev' > b'nh_netdev_event' > b'raw_notifier_call_chain' > b'call_netdevice_notifiers_info' > b'__dev_notify_flags' > b'dev_change_flags' > b'do_setlink' > b'__rtnl_newlink' > b'rtnl_newlink' > b'rtnetlink_rcv_msg' > b'netlink_rcv_skb' > b'rtnetlink_rcv' > b'netlink_unicast' > b'netlink_sendmsg' > b'____sys_sendmsg' > b'___sys_sendmsg' > b'__sys_sendmsg' > b'__x64_sys_sendmsg' > b'do_syscall_64' > b'entry_SYSCALL_64_after_hwframe' > - ip (277) > 126554955 > > Since nexthops are always deleted under RTNL, synchronize_net() can be > used instead. It will call synchronize_rcu_expedited() which only blocks > for several microseconds as opposed to multiple milliseconds like > synchronize_rcu(). > > With this patch deletion of 16k nexthops takes less than a second: > > # time -p ip link set dev dummy10 down > real 0.12 > user 0.00 > sys 0.04 > > Tested with fib_nexthops.sh which includes torture tests that prompted > the initial change: > > # ./fib_nexthops.sh > ... > Tests passed: 134 > Tests failed: 0 > > Fixes: 90f33bffa382 ("nexthops: don't modify published nexthop groups") > Signed-off-by: Ido Schimmel > --- > net/ipv4/nexthop.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > Thanks for finding this, Ido. Reviewed-by: David Ahern