From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.17.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2F1EEC4 for ; Wed, 25 May 2022 14:08:28 +0000 (UTC) Received: from [192.168.1.107] ([37.4.249.139]) by mrelayeu.kundenserver.de (mreue106 [212.227.15.183]) with ESMTPSA (Nemesis) id 1Mgf4k-1nLspk43qO-00h8Jk; Wed, 25 May 2022 16:07:49 +0200 Message-ID: Date: Wed, 25 May 2022 16:07:47 +0200 Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: vchiq: Performance regression since 5.18-rc1 Content-Language: en-US To: Marcelo Tosatti , Sebastian Andrzej Siewior Cc: Andrew Morton , Nicolas Saenz Julienne , Borislav Petkov , Minchan Kim , Matthew Wilcox , Mel Gorman , Juri Lelli , Thomas Gleixner , "Paul E. McKenney" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linux ARM , Phil Elwell , regressions@lists.linux.dev References: <77d6d498-7dd9-03eb-60f2-d7e682bb1b20@i2se.com> From: Stefan Wahren In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:p2z5Oxmov4jctz2aZQ5+CFweQ9+ZNhzGKPD1mZA7B+QEblEEdaD tlvpMKuRiEtGObW2wGBheYFDje5vsZm5tBrRl2bgl4SpDQDPy8FY4RoGr14WGSFwdJ2JLwx A7J43bqplqke3y0uMcbn0qEjktj9Z4LEG4CVryOnQue3lviFw0hD44zqB6AYPVPAHG0sfpJ IageqzzK3P6j5syqUtVaw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:kNIYU9kHu2I=:LIg2vrOd5uFgdnHurX25Ir qwu1g5ExP0CzOALXneYaDKz7c9W5XYVgvx5Ggx1OF2WpyRE+Vcym1HIIyIkyXAS941MxtHknw zVWsXJELOA0gJnk/tCEGg/MVW1k9LB9i5sAwsbanBObp7Lmv7azPx+dbOg91+gC57pjTMxOSe QE/VOmR2aFnM0WcjPTtYI8bq3+PjoxtPwb4zv4H4USIDD+N9dU8rmPdOqEdOID5m67ue9zHnW aKSvEj/CQZhDE+DH8SAp1qWpqEH5CQsMdJ1ESp56YxPVBZTOsbyTe3gt55RAb31Tpg+/QyBON lgrUW+JYdkEebUZYa5ZZ7KpWUM8cZwtd1/RG7VD/1qxb8he0USHlz++ott8djTjeAeohY5zvL CmwTCi/iCmBo6vRQ44h+a7RXzC0U5XimyHY1fDh3e+zdqK+6vcZ6yZ2n7CmuV8ibQyjzD6LPg 8kBiv2w2ZQgPX/NkfiiKrIoi9d6NuaO4dru2xT75rBjUblv1HBjC5PfcFJbk/AgV6zPjiyobJ uZLGfngTpXajbdX1cjcKuAwcHwWsPfitWq/UuRx3ZkBZdHLQCFPuoThqPZkhMDaM4NjGoOr5v sjf+boyI28MP+9U8PtdTYTboUzlu2zyGrp5MZWBaHHNfcgETP7LsgLGo/yYZOD9F/tngEVayt 9YlBD+x3baMIjX5VJVosHvrGR2BqOgCxL5QkFYIr8PwKvE3PswD14HJljpKKqAi1a7MLvGnbT tYLXIjI1cIb+uogACIffw8pdlZG8APrAq4WvyXm27fVH40sxpvMGDoP/q2M= Hi Marcelo, Am 25.05.22 um 15:56 schrieb Marcelo Tosatti: > On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote: >> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote: >>> Hi, >> Hi, >> >>> while testing the staging/vc04_services/interface/vchiq_arm driver with my >>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance >>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm: >>> lru_cache_disable: replace work queue synchronization with synchronize_rcu >>> >>> Usually i run "vchiq_test -f 1" to see the driver is still working [1]. >> What about >> https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/ >> >> Sebastian > Stefan, > > Can you please try the patch above ? this was the same as Paul send. I think i need more time for investigation, maybe there is an issue with the application. All i noticed so far is that in good case the CPU usage is around ~ 60 % and higher, while in bad case the CPU is almost idle. Also the issue is not reproducible with arm64/defconfig. > > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 855E3C433FE for ; Wed, 25 May 2022 14:09:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lmL6Z3LlclYpEK/Rcl+i4mF6Rs2E0imqz9J23WzJqzc=; b=MLNiu7r21gICXO Wq5AvY7A3djNUnA7B2LeShVFKMeifDjJjOR71dRlNAZpdSCFYGN356ZBf7oC7Ak6ecqa53gfEexp3 jVLBuoFueLI4SY/zZ5JLCGYFoNY67IVAVQHwc1NE2SocVykzdPAWrnFks0OshxMSbA7v39ffnPKVQ mBvahPzL9CTQ9CMUcmQWzRCaZBzlc26gCQwlQyM36+u3Mcb3lvhlg2b/g3MW4rqN5rB0wvqe6paJG Knib8AeOT6lfbwYJb6PjzEucp9+cPOZ/FPiUOJLGrvlnaTWqwQJfviV9a0tcJUc56WJfr+5W5HySe QgaAWjP6QHXZSKZFc/6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ntrgR-00BO2k-H0; Wed, 25 May 2022 14:08:19 +0000 Received: from mout.kundenserver.de ([212.227.17.24]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ntrgN-00BO1b-Vb for linux-arm-kernel@lists.infradead.org; Wed, 25 May 2022 14:08:17 +0000 Received: from [192.168.1.107] ([37.4.249.139]) by mrelayeu.kundenserver.de (mreue106 [212.227.15.183]) with ESMTPSA (Nemesis) id 1Mgf4k-1nLspk43qO-00h8Jk; Wed, 25 May 2022 16:07:49 +0200 Message-ID: Date: Wed, 25 May 2022 16:07:47 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: vchiq: Performance regression since 5.18-rc1 Content-Language: en-US To: Marcelo Tosatti , Sebastian Andrzej Siewior Cc: Andrew Morton , Nicolas Saenz Julienne , Borislav Petkov , Minchan Kim , Matthew Wilcox , Mel Gorman , Juri Lelli , Thomas Gleixner , "Paul E. McKenney" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linux ARM , Phil Elwell , regressions@lists.linux.dev References: <77d6d498-7dd9-03eb-60f2-d7e682bb1b20@i2se.com> From: Stefan Wahren In-Reply-To: X-Provags-ID: V03:K1:p2z5Oxmov4jctz2aZQ5+CFweQ9+ZNhzGKPD1mZA7B+QEblEEdaD tlvpMKuRiEtGObW2wGBheYFDje5vsZm5tBrRl2bgl4SpDQDPy8FY4RoGr14WGSFwdJ2JLwx A7J43bqplqke3y0uMcbn0qEjktj9Z4LEG4CVryOnQue3lviFw0hD44zqB6AYPVPAHG0sfpJ IageqzzK3P6j5syqUtVaw== X-UI-Out-Filterresults: notjunk:1;V03:K0:kNIYU9kHu2I=:LIg2vrOd5uFgdnHurX25Ir qwu1g5ExP0CzOALXneYaDKz7c9W5XYVgvx5Ggx1OF2WpyRE+Vcym1HIIyIkyXAS941MxtHknw zVWsXJELOA0gJnk/tCEGg/MVW1k9LB9i5sAwsbanBObp7Lmv7azPx+dbOg91+gC57pjTMxOSe QE/VOmR2aFnM0WcjPTtYI8bq3+PjoxtPwb4zv4H4USIDD+N9dU8rmPdOqEdOID5m67ue9zHnW aKSvEj/CQZhDE+DH8SAp1qWpqEH5CQsMdJ1ESp56YxPVBZTOsbyTe3gt55RAb31Tpg+/QyBON lgrUW+JYdkEebUZYa5ZZ7KpWUM8cZwtd1/RG7VD/1qxb8he0USHlz++ott8djTjeAeohY5zvL CmwTCi/iCmBo6vRQ44h+a7RXzC0U5XimyHY1fDh3e+zdqK+6vcZ6yZ2n7CmuV8ibQyjzD6LPg 8kBiv2w2ZQgPX/NkfiiKrIoi9d6NuaO4dru2xT75rBjUblv1HBjC5PfcFJbk/AgV6zPjiyobJ uZLGfngTpXajbdX1cjcKuAwcHwWsPfitWq/UuRx3ZkBZdHLQCFPuoThqPZkhMDaM4NjGoOr5v sjf+boyI28MP+9U8PtdTYTboUzlu2zyGrp5MZWBaHHNfcgETP7LsgLGo/yYZOD9F/tngEVayt 9YlBD+x3baMIjX5VJVosHvrGR2BqOgCxL5QkFYIr8PwKvE3PswD14HJljpKKqAi1a7MLvGnbT tYLXIjI1cIb+uogACIffw8pdlZG8APrAq4WvyXm27fVH40sxpvMGDoP/q2M= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220525_070816_366220_7BAEBDEB X-CRM114-Status: GOOD ( 17.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Marcelo, Am 25.05.22 um 15:56 schrieb Marcelo Tosatti: > On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote: >> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote: >>> Hi, >> Hi, >> >>> while testing the staging/vc04_services/interface/vchiq_arm driver with my >>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance >>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm: >>> lru_cache_disable: replace work queue synchronization with synchronize_rcu >>> >>> Usually i run "vchiq_test -f 1" to see the driver is still working [1]. >> What about >> https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/ >> >> Sebastian > Stefan, > > Can you please try the patch above ? this was the same as Paul send. I think i need more time for investigation, maybe there is an issue with the application. All i noticed so far is that in good case the CPU usage is around ~ 60 % and higher, while in bad case the CPU is almost idle. Also the issue is not reproducible with arm64/defconfig. > > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel