vchiq: Performance regression since 5.18-rc1

vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi,

while testing the staging/vc04_services/interface/vchiq_arm driver with 
my Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance 
regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm: 
lru_cache_disable: replace work queue synchronization with synchronize_rcu

Usually i run "vchiq_test -f 1" to see the driver is still working [1].

Before commit:

real    0m1,500s
user    0m0,068s
sys    0m0,846s

After commit:

real    7m11,449s
user    0m2,049s
sys    0m0,023s

Best regards

[1] - https://github.com/raspberrypi/userland

vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi,

while testing the staging/vc04_services/interface/vchiq_arm driver with 
my Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance 
regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm: 
lru_cache_disable: replace work queue synchronization with synchronize_rcu

Usually i run "vchiq_test -f 1" to see the driver is still working [1].

Before commit:

real    0m1,500s
user    0m0,068s
sys    0m0,846s

After commit:

real    7m11,449s
user    0m2,049s
sys    0m0,023s

Best regards

[1] - https://github.com/raspberrypi/userland



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Paul E. McKenney]

On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
> Hi,
> 
> while testing the staging/vc04_services/interface/vchiq_arm driver with my
> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> 
> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> 
> Before commit:
> 
> real    0m1,500s
> user    0m0,068s
> sys    0m0,846s
> 
> After commit:
> 
> real    7m11,449s
> user    0m2,049s
> sys    0m0,023s
> 
> Best regards
> 
> [1] - https://github.com/raspberrypi/userland

Please feel free to try the patch shown below.  Or the pair of patches
from Rik here:

https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/

There is work ongoing to produce something better, but ongoing slowly.
Especially my part of that work.

							Thanx, Paul

------------------------------------------------------------------------

From paulmck@kernel.org Mon Feb 14 11:05:49 2022
Date: Mon, 14 Feb 2022 11:05:49 -0800
From: "Paul E. McKenney" <paulmck@kernel.org>
To: clm@fb.com
Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
 synchronize_rcu_expedited()
Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
Reply-To: paulmck@kernel.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Status: RO
Content-Length: 1036
Lines: 32

Experimental.  Not for inclusion.  Yet, anyway.

Freeing large numbers of namespaces in quick succession can result in
a bottleneck on the synchronize_rcu() invoked from kern_unmount().
This patch applies the synchronize_rcu_expedited() hammer to allow
further testing and fault isolation.

Hey, at least there was no need to change the comment!  ;-)

Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: <linux-fsdevel@vger.kernel.org>
Cc: <linux-kernel@vger.kernel.org>
Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>

---

 namespace.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index 40b994a29e90d..79c50ad0ade5b 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
 	/* release long term mount so mount point can be released */
 	if (!IS_ERR_OR_NULL(mnt)) {
 		real_mount(mnt)->mnt_ns = NULL;
-		synchronize_rcu();	/* yecchhh... */
+		synchronize_rcu_expedited();	/* yecchhh... */
 		mntput(mnt);
 	}
 }

Re: vchiq: Performance regression since 5.18-rc1

[Paul E. McKenney]

On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
> Hi,
> 
> while testing the staging/vc04_services/interface/vchiq_arm driver with my
> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> 
> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> 
> Before commit:
> 
> real    0m1,500s
> user    0m0,068s
> sys    0m0,846s
> 
> After commit:
> 
> real    7m11,449s
> user    0m2,049s
> sys    0m0,023s
> 
> Best regards
> 
> [1] - https://github.com/raspberrypi/userland

Please feel free to try the patch shown below.  Or the pair of patches
from Rik here:

https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/

There is work ongoing to produce something better, but ongoing slowly.
Especially my part of that work.

							Thanx, Paul

------------------------------------------------------------------------

From paulmck@kernel.org Mon Feb 14 11:05:49 2022
Date: Mon, 14 Feb 2022 11:05:49 -0800
From: "Paul E. McKenney" <paulmck@kernel.org>
To: clm@fb.com
Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
 synchronize_rcu_expedited()
Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
Reply-To: paulmck@kernel.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Status: RO
Content-Length: 1036
Lines: 32

Experimental.  Not for inclusion.  Yet, anyway.

Freeing large numbers of namespaces in quick succession can result in
a bottleneck on the synchronize_rcu() invoked from kern_unmount().
This patch applies the synchronize_rcu_expedited() hammer to allow
further testing and fault isolation.

Hey, at least there was no need to change the comment!  ;-)

Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: <linux-fsdevel@vger.kernel.org>
Cc: <linux-kernel@vger.kernel.org>
Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>

---

 namespace.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index 40b994a29e90d..79c50ad0ade5b 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
 	/* release long term mount so mount point can be released */
 	if (!IS_ERR_OR_NULL(mnt)) {
 		real_mount(mnt)->mnt_ns = NULL;
-		synchronize_rcu();	/* yecchhh... */
+		synchronize_rcu_expedited();	/* yecchhh... */
 		mntput(mnt);
 	}
 }


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Paul,

Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>> Hi,
>>
>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>
>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>
>> Before commit:
>>
>> real    0m1,500s
>> user    0m0,068s
>> sys    0m0,846s
>>
>> After commit:
>>
>> real    7m11,449s
>> user    0m2,049s
>> sys    0m0,023s
>>
>> Best regards
>>
>> [1] - https://github.com/raspberrypi/userland
> Please feel free to try the patch shown below.  Or the pair of patches
> from Rik here:
>
> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/

I tried your patch and Rik's patches but in both cases vchiq_test runs 7 
minutes instead of ~ 1 second.

Best regards

>
> There is work ongoing to produce something better, but ongoing slowly.
> Especially my part of that work.
>
> 							Thanx, Paul
>
> ------------------------------------------------------------------------
>
>  From paulmck@kernel.org Mon Feb 14 11:05:49 2022
> Date: Mon, 14 Feb 2022 11:05:49 -0800
> From: "Paul E. McKenney" <paulmck@kernel.org>
> To: clm@fb.com
> Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
> 	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
> Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
>   synchronize_rcu_expedited()
> Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
> Reply-To: paulmck@kernel.org
> MIME-Version: 1.0
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> Status: RO
> Content-Length: 1036
> Lines: 32
>
> Experimental.  Not for inclusion.  Yet, anyway.
>
> Freeing large numbers of namespaces in quick succession can result in
> a bottleneck on the synchronize_rcu() invoked from kern_unmount().
> This patch applies the synchronize_rcu_expedited() hammer to allow
> further testing and fault isolation.
>
> Hey, at least there was no need to change the comment!  ;-)
>
> Cc: Alexander Viro <viro@zeniv.linux.org.uk>
> Cc: <linux-fsdevel@vger.kernel.org>
> Cc: <linux-kernel@vger.kernel.org>
> Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>
> ---
>
>   namespace.c |    2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/namespace.c b/fs/namespace.c
> index 40b994a29e90d..79c50ad0ade5b 100644
> --- a/fs/namespace.c
> +++ b/fs/namespace.c
> @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
>   	/* release long term mount so mount point can be released */
>   	if (!IS_ERR_OR_NULL(mnt)) {
>   		real_mount(mnt)->mnt_ns = NULL;
> -		synchronize_rcu();	/* yecchhh... */
> +		synchronize_rcu_expedited();	/* yecchhh... */
>   		mntput(mnt);
>   	}
>   }
>

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Paul,

Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>> Hi,
>>
>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>
>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>
>> Before commit:
>>
>> real    0m1,500s
>> user    0m0,068s
>> sys    0m0,846s
>>
>> After commit:
>>
>> real    7m11,449s
>> user    0m2,049s
>> sys    0m0,023s
>>
>> Best regards
>>
>> [1] - https://github.com/raspberrypi/userland
> Please feel free to try the patch shown below.  Or the pair of patches
> from Rik here:
>
> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/

I tried your patch and Rik's patches but in both cases vchiq_test runs 7 
minutes instead of ~ 1 second.

Best regards

>
> There is work ongoing to produce something better, but ongoing slowly.
> Especially my part of that work.
>
> 							Thanx, Paul
>
> ------------------------------------------------------------------------
>
>  From paulmck@kernel.org Mon Feb 14 11:05:49 2022
> Date: Mon, 14 Feb 2022 11:05:49 -0800
> From: "Paul E. McKenney" <paulmck@kernel.org>
> To: clm@fb.com
> Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
> 	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
> Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
>   synchronize_rcu_expedited()
> Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
> Reply-To: paulmck@kernel.org
> MIME-Version: 1.0
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> Status: RO
> Content-Length: 1036
> Lines: 32
>
> Experimental.  Not for inclusion.  Yet, anyway.
>
> Freeing large numbers of namespaces in quick succession can result in
> a bottleneck on the synchronize_rcu() invoked from kern_unmount().
> This patch applies the synchronize_rcu_expedited() hammer to allow
> further testing and fault isolation.
>
> Hey, at least there was no need to change the comment!  ;-)
>
> Cc: Alexander Viro <viro@zeniv.linux.org.uk>
> Cc: <linux-fsdevel@vger.kernel.org>
> Cc: <linux-kernel@vger.kernel.org>
> Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>
> ---
>
>   namespace.c |    2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/namespace.c b/fs/namespace.c
> index 40b994a29e90d..79c50ad0ade5b 100644
> --- a/fs/namespace.c
> +++ b/fs/namespace.c
> @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
>   	/* release long term mount so mount point can be released */
>   	if (!IS_ERR_OR_NULL(mnt)) {
>   		real_mount(mnt)->mnt_ns = NULL;
> -		synchronize_rcu();	/* yecchhh... */
> +		synchronize_rcu_expedited();	/* yecchhh... */
>   		mntput(mnt);
>   	}
>   }
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Paul E. McKenney]

On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
> Hi Paul,
> 
> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
> > On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
> > > Hi,
> > > 
> > > while testing the staging/vc04_services/interface/vchiq_arm driver with my
> > > Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> > > regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> > > lru_cache_disable: replace work queue synchronization with synchronize_rcu
> > > 
> > > Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> > > 
> > > Before commit:
> > > 
> > > real    0m1,500s
> > > user    0m0,068s
> > > sys    0m0,846s
> > > 
> > > After commit:
> > > 
> > > real    7m11,449s
> > > user    0m2,049s
> > > sys    0m0,023s
> > > 
> > > Best regards
> > > 
> > > [1] - https://github.com/raspberrypi/userland
> > Please feel free to try the patch shown below.  Or the pair of patches
> > from Rik here:
> > 
> > https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
> > https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
> 
> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
> minutes instead of ~ 1 second.

That is surprising.  Do you boot with rcupdate.rcu_normal=1?  That would
nullify my patch, but I would expect that Rik's patch would still provide
increased performance even in that case.

Could you please characterize where the slowdown is occurring?

							Thanx, Paul

> Best regards
> 
> > 
> > There is work ongoing to produce something better, but ongoing slowly.
> > Especially my part of that work.
> > 
> > 							Thanx, Paul
> > 
> > ------------------------------------------------------------------------
> > 
> >  From paulmck@kernel.org Mon Feb 14 11:05:49 2022
> > Date: Mon, 14 Feb 2022 11:05:49 -0800
> > From: "Paul E. McKenney" <paulmck@kernel.org>
> > To: clm@fb.com
> > Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
> > 	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
> > Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
> >   synchronize_rcu_expedited()
> > Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
> > Reply-To: paulmck@kernel.org
> > MIME-Version: 1.0
> > Content-Type: text/plain; charset=us-ascii
> > Content-Disposition: inline
> > Status: RO
> > Content-Length: 1036
> > Lines: 32
> > 
> > Experimental.  Not for inclusion.  Yet, anyway.
> > 
> > Freeing large numbers of namespaces in quick succession can result in
> > a bottleneck on the synchronize_rcu() invoked from kern_unmount().
> > This patch applies the synchronize_rcu_expedited() hammer to allow
> > further testing and fault isolation.
> > 
> > Hey, at least there was no need to change the comment!  ;-)
> > 
> > Cc: Alexander Viro <viro@zeniv.linux.org.uk>
> > Cc: <linux-fsdevel@vger.kernel.org>
> > Cc: <linux-kernel@vger.kernel.org>
> > Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > 
> > ---
> > 
> >   namespace.c |    2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/fs/namespace.c b/fs/namespace.c
> > index 40b994a29e90d..79c50ad0ade5b 100644
> > --- a/fs/namespace.c
> > +++ b/fs/namespace.c
> > @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
> >   	/* release long term mount so mount point can be released */
> >   	if (!IS_ERR_OR_NULL(mnt)) {
> >   		real_mount(mnt)->mnt_ns = NULL;
> > -		synchronize_rcu();	/* yecchhh... */
> > +		synchronize_rcu_expedited();	/* yecchhh... */
> >   		mntput(mnt);
> >   	}
> >   }
> > 

Re: vchiq: Performance regression since 5.18-rc1

[Paul E. McKenney]

On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
> Hi Paul,
> 
> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
> > On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
> > > Hi,
> > > 
> > > while testing the staging/vc04_services/interface/vchiq_arm driver with my
> > > Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> > > regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> > > lru_cache_disable: replace work queue synchronization with synchronize_rcu
> > > 
> > > Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> > > 
> > > Before commit:
> > > 
> > > real    0m1,500s
> > > user    0m0,068s
> > > sys    0m0,846s
> > > 
> > > After commit:
> > > 
> > > real    7m11,449s
> > > user    0m2,049s
> > > sys    0m0,023s
> > > 
> > > Best regards
> > > 
> > > [1] - https://github.com/raspberrypi/userland
> > Please feel free to try the patch shown below.  Or the pair of patches
> > from Rik here:
> > 
> > https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
> > https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
> 
> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
> minutes instead of ~ 1 second.

That is surprising.  Do you boot with rcupdate.rcu_normal=1?  That would
nullify my patch, but I would expect that Rik's patch would still provide
increased performance even in that case.

Could you please characterize where the slowdown is occurring?

							Thanx, Paul

> Best regards
> 
> > 
> > There is work ongoing to produce something better, but ongoing slowly.
> > Especially my part of that work.
> > 
> > 							Thanx, Paul
> > 
> > ------------------------------------------------------------------------
> > 
> >  From paulmck@kernel.org Mon Feb 14 11:05:49 2022
> > Date: Mon, 14 Feb 2022 11:05:49 -0800
> > From: "Paul E. McKenney" <paulmck@kernel.org>
> > To: clm@fb.com
> > Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
> > 	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
> > Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
> >   synchronize_rcu_expedited()
> > Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
> > Reply-To: paulmck@kernel.org
> > MIME-Version: 1.0
> > Content-Type: text/plain; charset=us-ascii
> > Content-Disposition: inline
> > Status: RO
> > Content-Length: 1036
> > Lines: 32
> > 
> > Experimental.  Not for inclusion.  Yet, anyway.
> > 
> > Freeing large numbers of namespaces in quick succession can result in
> > a bottleneck on the synchronize_rcu() invoked from kern_unmount().
> > This patch applies the synchronize_rcu_expedited() hammer to allow
> > further testing and fault isolation.
> > 
> > Hey, at least there was no need to change the comment!  ;-)
> > 
> > Cc: Alexander Viro <viro@zeniv.linux.org.uk>
> > Cc: <linux-fsdevel@vger.kernel.org>
> > Cc: <linux-kernel@vger.kernel.org>
> > Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > 
> > ---
> > 
> >   namespace.c |    2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/fs/namespace.c b/fs/namespace.c
> > index 40b994a29e90d..79c50ad0ade5b 100644
> > --- a/fs/namespace.c
> > +++ b/fs/namespace.c
> > @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
> >   	/* release long term mount so mount point can be released */
> >   	if (!IS_ERR_OR_NULL(mnt)) {
> >   		real_mount(mnt)->mnt_ns = NULL;
> > -		synchronize_rcu();	/* yecchhh... */
> > +		synchronize_rcu_expedited();	/* yecchhh... */
> >   		mntput(mnt);
> >   	}
> >   }
> > 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Paul,

Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>> Hi Paul,
>>
>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>> Hi,
>>>>
>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>
>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>
>>>> Before commit:
>>>>
>>>> real    0m1,500s
>>>> user    0m0,068s
>>>> sys    0m0,846s
>>>>
>>>> After commit:
>>>>
>>>> real    7m11,449s
>>>> user    0m2,049s
>>>> sys    0m0,023s
>>>>
>>>> Best regards
>>>>
>>>> [1] - https://github.com/raspberrypi/userland
>>> Please feel free to try the patch shown below.  Or the pair of patches
>>> from Rik here:
>>>
>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>> minutes instead of ~ 1 second.
> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
No, not explicit.
>    That would
> nullify my patch, but I would expect that Rik's patch would still provide
> increased performance even in that case.
I will retest with a fresh SD card image.
>
> Could you please characterize where the slowdown is occurring?

Unfortunately i don't have a deep insight into driver and vchiq_test 
tool. Just a user view.

Do you think an strace would be a good starting point?

@Phil Any advices to analyse this issue?

>
> 							Thanx, Paul
>
>> Best regards
>>
>>> There is work ongoing to produce something better, but ongoing slowly.
>>> Especially my part of that work.
>>>
>>> 							Thanx, Paul
>>>
>>> ------------------------------------------------------------------------
>>>
>>>   From paulmck@kernel.org Mon Feb 14 11:05:49 2022
>>> Date: Mon, 14 Feb 2022 11:05:49 -0800
>>> From: "Paul E. McKenney" <paulmck@kernel.org>
>>> To: clm@fb.com
>>> Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
>>> 	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
>>> Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
>>>    synchronize_rcu_expedited()
>>> Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
>>> Reply-To: paulmck@kernel.org
>>> MIME-Version: 1.0
>>> Content-Type: text/plain; charset=us-ascii
>>> Content-Disposition: inline
>>> Status: RO
>>> Content-Length: 1036
>>> Lines: 32
>>>
>>> Experimental.  Not for inclusion.  Yet, anyway.
>>>
>>> Freeing large numbers of namespaces in quick succession can result in
>>> a bottleneck on the synchronize_rcu() invoked from kern_unmount().
>>> This patch applies the synchronize_rcu_expedited() hammer to allow
>>> further testing and fault isolation.
>>>
>>> Hey, at least there was no need to change the comment!  ;-)
>>>
>>> Cc: Alexander Viro <viro@zeniv.linux.org.uk>
>>> Cc: <linux-fsdevel@vger.kernel.org>
>>> Cc: <linux-kernel@vger.kernel.org>
>>> Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>>>
>>> ---
>>>
>>>    namespace.c |    2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/fs/namespace.c b/fs/namespace.c
>>> index 40b994a29e90d..79c50ad0ade5b 100644
>>> --- a/fs/namespace.c
>>> +++ b/fs/namespace.c
>>> @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
>>>    	/* release long term mount so mount point can be released */
>>>    	if (!IS_ERR_OR_NULL(mnt)) {
>>>    		real_mount(mnt)->mnt_ns = NULL;
>>> -		synchronize_rcu();	/* yecchhh... */
>>> +		synchronize_rcu_expedited();	/* yecchhh... */
>>>    		mntput(mnt);
>>>    	}
>>>    }
>>>

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Paul,

Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>> Hi Paul,
>>
>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>> Hi,
>>>>
>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>
>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>
>>>> Before commit:
>>>>
>>>> real    0m1,500s
>>>> user    0m0,068s
>>>> sys    0m0,846s
>>>>
>>>> After commit:
>>>>
>>>> real    7m11,449s
>>>> user    0m2,049s
>>>> sys    0m0,023s
>>>>
>>>> Best regards
>>>>
>>>> [1] - https://github.com/raspberrypi/userland
>>> Please feel free to try the patch shown below.  Or the pair of patches
>>> from Rik here:
>>>
>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>> minutes instead of ~ 1 second.
> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
No, not explicit.
>    That would
> nullify my patch, but I would expect that Rik's patch would still provide
> increased performance even in that case.
I will retest with a fresh SD card image.
>
> Could you please characterize where the slowdown is occurring?

Unfortunately i don't have a deep insight into driver and vchiq_test 
tool. Just a user view.

Do you think an strace would be a good starting point?

@Phil Any advices to analyse this issue?

>
> 							Thanx, Paul
>
>> Best regards
>>
>>> There is work ongoing to produce something better, but ongoing slowly.
>>> Especially my part of that work.
>>>
>>> 							Thanx, Paul
>>>
>>> ------------------------------------------------------------------------
>>>
>>>   From paulmck@kernel.org Mon Feb 14 11:05:49 2022
>>> Date: Mon, 14 Feb 2022 11:05:49 -0800
>>> From: "Paul E. McKenney" <paulmck@kernel.org>
>>> To: clm@fb.com
>>> Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org,
>>> 	linux-fsdevel@vger.kernel.org, kernel-team@fb.com
>>> Subject: [PATCH RFC fs/namespace] Make kern_unmount() use
>>>    synchronize_rcu_expedited()
>>> Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1>
>>> Reply-To: paulmck@kernel.org
>>> MIME-Version: 1.0
>>> Content-Type: text/plain; charset=us-ascii
>>> Content-Disposition: inline
>>> Status: RO
>>> Content-Length: 1036
>>> Lines: 32
>>>
>>> Experimental.  Not for inclusion.  Yet, anyway.
>>>
>>> Freeing large numbers of namespaces in quick succession can result in
>>> a bottleneck on the synchronize_rcu() invoked from kern_unmount().
>>> This patch applies the synchronize_rcu_expedited() hammer to allow
>>> further testing and fault isolation.
>>>
>>> Hey, at least there was no need to change the comment!  ;-)
>>>
>>> Cc: Alexander Viro <viro@zeniv.linux.org.uk>
>>> Cc: <linux-fsdevel@vger.kernel.org>
>>> Cc: <linux-kernel@vger.kernel.org>
>>> Not-yet-signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>>>
>>> ---
>>>
>>>    namespace.c |    2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/fs/namespace.c b/fs/namespace.c
>>> index 40b994a29e90d..79c50ad0ade5b 100644
>>> --- a/fs/namespace.c
>>> +++ b/fs/namespace.c
>>> @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt)
>>>    	/* release long term mount so mount point can be released */
>>>    	if (!IS_ERR_OR_NULL(mnt)) {
>>>    		real_mount(mnt)->mnt_ns = NULL;
>>> -		synchronize_rcu();	/* yecchhh... */
>>> +		synchronize_rcu_expedited();	/* yecchhh... */
>>>    		mntput(mnt);
>>>    	}
>>>    }
>>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Sebastian Andrzej Siewior]

On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
> Hi,
Hi,

> while testing the staging/vc04_services/interface/vchiq_arm driver with my
> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> 
> Usually i run "vchiq_test -f 1" to see the driver is still working [1].

What about
	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/

Sebastian

Re: vchiq: Performance regression since 5.18-rc1

[Sebastian Andrzej Siewior]

On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
> Hi,
Hi,

> while testing the staging/vc04_services/interface/vchiq_arm driver with my
> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> 
> Usually i run "vchiq_test -f 1" to see the driver is still working [1].

What about
	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/

Sebastian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Thorsten Leemhuis]

[TLDR: I'm adding this regression report to the list of tracked
regressions; all text from me you find below is based on a few templates
paragraphs you might have encountered already already in similar form.]

Hi, this is your Linux kernel regression tracker.

On 22.05.22 01:22, Stefan Wahren wrote:
> 
> while testing the staging/vc04_services/interface/vchiq_arm driver with
> my Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> 
> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> 
> Before commit:
> 
> real    0m1,500s
> user    0m0,068s
> sys    0m0,846s
> 
> After commit:
> 
> real    7m11,449s
> user    0m2,049s
> sys    0m0,023s

Thanks for the report.

To be sure below issue doesn't fall through the cracks unnoticed, I'm
adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced ff042f4a9b050895a42cae893cc01fa2ca81b95
#regzbot title mm: chiq_test runs 7 minutes instead of ~ 1 second.
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply -- ideally with also
telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (the mail this one replied to), as the kernel's
documentation call for; above page explains why this is important for
tracked regressions.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Re: vchiq: Performance regression since 5.18-rc1

[Thorsten Leemhuis]

[TLDR: I'm adding this regression report to the list of tracked
regressions; all text from me you find below is based on a few templates
paragraphs you might have encountered already already in similar form.]

Hi, this is your Linux kernel regression tracker.

On 22.05.22 01:22, Stefan Wahren wrote:
> 
> while testing the staging/vc04_services/interface/vchiq_arm driver with
> my Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> 
> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> 
> Before commit:
> 
> real    0m1,500s
> user    0m0,068s
> sys    0m0,846s
> 
> After commit:
> 
> real    7m11,449s
> user    0m2,049s
> sys    0m0,023s

Thanks for the report.

To be sure below issue doesn't fall through the cracks unnoticed, I'm
adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced ff042f4a9b050895a42cae893cc01fa2ca81b95
#regzbot title mm: chiq_test runs 7 minutes instead of ~ 1 second.
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply -- ideally with also
telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (the mail this one replied to), as the kernel's
documentation call for; above page explains why this is important for
tracked regressions.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Phil Elwell]

Hi Stefan,

On 23/05/2022 07:19, Stefan Wahren wrote:
> Hi Paul,
> 
> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>> Hi Paul,
>>>
>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>> Hi,
>>>>>
>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>
>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>>
>>>>> Before commit:
>>>>>
>>>>> real    0m1,500s
>>>>> user    0m0,068s
>>>>> sys    0m0,846s
>>>>>
>>>>> After commit:
>>>>>
>>>>> real    7m11,449s
>>>>> user    0m2,049s
>>>>> sys    0m0,023s
>>>>>
>>>>> Best regards
>>>>>
>>>>> [1] - https://github.com/raspberrypi/userland
>>>> Please feel free to try the patch shown below.  Or the pair of patches
>>>> from Rik here:
>>>>
>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>>> minutes instead of ~ 1 second.
>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
> No, not explicit.
>>    That would
>> nullify my patch, but I would expect that Rik's patch would still provide
>> increased performance even in that case.
> I will retest with a fresh SD card image.
>>
>> Could you please characterize where the slowdown is occurring?
> 
> Unfortunately i don't have a deep insight into driver and vchiq_test tool. Just 
> a user view.
> 
> Do you think an strace would be a good starting point?
> 
> @Phil Any advices to analyse this issue?

Sending many small control packets:

    vchiq_test -c 1 10000

essentially tests interrupt latency. Using a small number of large bulk transfers:

    vchiq_test -b 10000 1

becomes a test of how long it takes to lock down pages. It also tests DMA 
transfer speeds, but since the DMA is run by the firmware (which you aren't 
changing), I think you can rule that.

You may also find it helpful to include "force_turbo=1" in config.txt for more 
predictable results.

By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any 
performance problems:

pi@raspberrypi:~$ time vchiq_test -f 1
Functional test - iters:1
======== iteration 1 ========
Testing bulk transfer for alignment.
Testing bulk transfer at PAGE_SIZE.

real    0m0.512s
user    0m0.042s
sys     0m0.165s

Phil

Re: vchiq: Performance regression since 5.18-rc1

[Phil Elwell]

Hi Stefan,

On 23/05/2022 07:19, Stefan Wahren wrote:
> Hi Paul,
> 
> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>> Hi Paul,
>>>
>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>> Hi,
>>>>>
>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>
>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>>
>>>>> Before commit:
>>>>>
>>>>> real    0m1,500s
>>>>> user    0m0,068s
>>>>> sys    0m0,846s
>>>>>
>>>>> After commit:
>>>>>
>>>>> real    7m11,449s
>>>>> user    0m2,049s
>>>>> sys    0m0,023s
>>>>>
>>>>> Best regards
>>>>>
>>>>> [1] - https://github.com/raspberrypi/userland
>>>> Please feel free to try the patch shown below.  Or the pair of patches
>>>> from Rik here:
>>>>
>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>>> minutes instead of ~ 1 second.
>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
> No, not explicit.
>>    That would
>> nullify my patch, but I would expect that Rik's patch would still provide
>> increased performance even in that case.
> I will retest with a fresh SD card image.
>>
>> Could you please characterize where the slowdown is occurring?
> 
> Unfortunately i don't have a deep insight into driver and vchiq_test tool. Just 
> a user view.
> 
> Do you think an strace would be a good starting point?
> 
> @Phil Any advices to analyse this issue?

Sending many small control packets:

    vchiq_test -c 1 10000

essentially tests interrupt latency. Using a small number of large bulk transfers:

    vchiq_test -b 10000 1

becomes a test of how long it takes to lock down pages. It also tests DMA 
transfer speeds, but since the DMA is run by the firmware (which you aren't 
changing), I think you can rule that.

You may also find it helpful to include "force_turbo=1" in config.txt for more 
predictable results.

By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any 
performance problems:

pi@raspberrypi:~$ time vchiq_test -f 1
Functional test - iters:1
======== iteration 1 ========
Testing bulk transfer for alignment.
Testing bulk transfer at PAGE_SIZE.

real    0m0.512s
user    0m0.042s
sys     0m0.165s

Phil

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Phil,

Am 23.05.22 um 11:29 schrieb Phil Elwell:
> Hi Stefan,
>
> On 23/05/2022 07:19, Stefan Wahren wrote:
>> Hi Paul,
>>
>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>> Hi Paul,
>>>>
>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>> Hi,
>>>>>>
>>>>>> while testing the staging/vc04_services/interface/vchiq_arm 
>>>>>> driver with my
>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>> lru_cache_disable: replace work queue synchronization with 
>>>>>> synchronize_rcu
>>>>>>
>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still 
>>>>>> working [1].
>>>>>>
>>>>>> Before commit:
>>>>>>
>>>>>> real    0m1,500s
>>>>>> user    0m0,068s
>>>>>> sys    0m0,846s
>>>>>>
>>>>>> After commit:
>>>>>>
>>>>>> real    7m11,449s
>>>>>> user    0m2,049s
>>>>>> sys    0m0,023s
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>> Please feel free to try the patch shown below.  Or the pair of 
>>>>> patches
>>>>> from Rik here:
>>>>>
>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/ 
>>>>>
>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/ 
>>>>>
>>>> I tried your patch and Rik's patches but in both cases vchiq_test 
>>>> runs 7
>>>> minutes instead of ~ 1 second.
>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>> No, not explicit.
>>>    That would
>>> nullify my patch, but I would expect that Rik's patch would still 
>>> provide
>>> increased performance even in that case.
>> I will retest with a fresh SD card image.
>>>
>>> Could you please characterize where the slowdown is occurring?
>>
>> Unfortunately i don't have a deep insight into driver and vchiq_test 
>> tool. Just a user view.
>>
>> Do you think an strace would be a good starting point?
>>
>> @Phil Any advices to analyse this issue?
>
> Sending many small control packets:
>
>    vchiq_test -c 1 10000
>
> essentially tests interrupt latency. Using a small number of large 
> bulk transfers:
>
>    vchiq_test -b 10000 1
>
> becomes a test of how long it takes to lock down pages. It also tests 
> DMA transfer speeds, but since the DMA is run by the firmware (which 
> you aren't changing), I think you can rule that.
Thanks i will try.
>
> You may also find it helpful to include "force_turbo=1" in config.txt 
> for more predictable results.
>
> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing 
> any performance problems:
I assume you are using arm/bcm2709_defconfig and not 
arm/multi_v7_defconfig as me?
>
> pi@raspberrypi:~$ time vchiq_test -f 1
> Functional test - iters:1
> ======== iteration 1 ========
> Testing bulk transfer for alignment.
> Testing bulk transfer at PAGE_SIZE.
>
> real    0m0.512s
> user    0m0.042s
> sys     0m0.165s
>
> Phil

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Phil,

Am 23.05.22 um 11:29 schrieb Phil Elwell:
> Hi Stefan,
>
> On 23/05/2022 07:19, Stefan Wahren wrote:
>> Hi Paul,
>>
>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>> Hi Paul,
>>>>
>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>> Hi,
>>>>>>
>>>>>> while testing the staging/vc04_services/interface/vchiq_arm 
>>>>>> driver with my
>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>> lru_cache_disable: replace work queue synchronization with 
>>>>>> synchronize_rcu
>>>>>>
>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still 
>>>>>> working [1].
>>>>>>
>>>>>> Before commit:
>>>>>>
>>>>>> real    0m1,500s
>>>>>> user    0m0,068s
>>>>>> sys    0m0,846s
>>>>>>
>>>>>> After commit:
>>>>>>
>>>>>> real    7m11,449s
>>>>>> user    0m2,049s
>>>>>> sys    0m0,023s
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>> Please feel free to try the patch shown below.  Or the pair of 
>>>>> patches
>>>>> from Rik here:
>>>>>
>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/ 
>>>>>
>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/ 
>>>>>
>>>> I tried your patch and Rik's patches but in both cases vchiq_test 
>>>> runs 7
>>>> minutes instead of ~ 1 second.
>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>> No, not explicit.
>>>    That would
>>> nullify my patch, but I would expect that Rik's patch would still 
>>> provide
>>> increased performance even in that case.
>> I will retest with a fresh SD card image.
>>>
>>> Could you please characterize where the slowdown is occurring?
>>
>> Unfortunately i don't have a deep insight into driver and vchiq_test 
>> tool. Just a user view.
>>
>> Do you think an strace would be a good starting point?
>>
>> @Phil Any advices to analyse this issue?
>
> Sending many small control packets:
>
>    vchiq_test -c 1 10000
>
> essentially tests interrupt latency. Using a small number of large 
> bulk transfers:
>
>    vchiq_test -b 10000 1
>
> becomes a test of how long it takes to lock down pages. It also tests 
> DMA transfer speeds, but since the DMA is run by the firmware (which 
> you aren't changing), I think you can rule that.
Thanks i will try.
>
> You may also find it helpful to include "force_turbo=1" in config.txt 
> for more predictable results.
>
> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing 
> any performance problems:
I assume you are using arm/bcm2709_defconfig and not 
arm/multi_v7_defconfig as me?
>
> pi@raspberrypi:~$ time vchiq_test -f 1
> Functional test - iters:1
> ======== iteration 1 ========
> Testing bulk transfer for alignment.
> Testing bulk transfer at PAGE_SIZE.
>
> real    0m0.512s
> user    0m0.042s
> sys     0m0.165s
>
> Phil

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Phil Elwell]

Hi Stefan,

On 23/05/2022 11:48, Stefan Wahren wrote:
> Hi Phil,
> 
> Am 23.05.22 um 11:29 schrieb Phil Elwell:
>> Hi Stefan,
>>
>> On 23/05/2022 07:19, Stefan Wahren wrote:
>>> Hi Paul,
>>>
>>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>>> Hi Paul,
>>>>>
>>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>>>
>>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>>>>
>>>>>>> Before commit:
>>>>>>>
>>>>>>> real    0m1,500s
>>>>>>> user    0m0,068s
>>>>>>> sys    0m0,846s
>>>>>>>
>>>>>>> After commit:
>>>>>>>
>>>>>>> real    7m11,449s
>>>>>>> user    0m2,049s
>>>>>>> sys    0m0,023s
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>>> Please feel free to try the patch shown below.  Or the pair of patches
>>>>>> from Rik here:
>>>>>>
>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>>>>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>>>>> minutes instead of ~ 1 second.
>>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>>> No, not explicit.
>>>>    That would
>>>> nullify my patch, but I would expect that Rik's patch would still provide
>>>> increased performance even in that case.
>>> I will retest with a fresh SD card image.
>>>>
>>>> Could you please characterize where the slowdown is occurring?
>>>
>>> Unfortunately i don't have a deep insight into driver and vchiq_test tool. 
>>> Just a user view.
>>>
>>> Do you think an strace would be a good starting point?
>>>
>>> @Phil Any advices to analyse this issue?
>>
>> Sending many small control packets:
>>
>>    vchiq_test -c 1 10000
>>
>> essentially tests interrupt latency. Using a small number of large bulk 
>> transfers:
>>
>>    vchiq_test -b 10000 1
>>
>> becomes a test of how long it takes to lock down pages. It also tests DMA 
>> transfer speeds, but since the DMA is run by the firmware (which you aren't 
>> changing), I think you can rule that.
> Thanks i will try.
>>
>> You may also find it helpful to include "force_turbo=1" in config.txt for more 
>> predictable results.
>>
>> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any 
>> performance problems:
> I assume you are using arm/bcm2709_defconfig and not arm/multi_v7_defconfig as me?

That's correct. Simply switching to multi_v7_defconfig breaks vchiq completely, 
presumably because it doesn't define CONFIG_BCM2835_VCHIQ.

Phil

>>
>> pi@raspberrypi:~$ time vchiq_test -f 1
>> Functional test - iters:1
>> ======== iteration 1 ========
>> Testing bulk transfer for alignment.
>> Testing bulk transfer at PAGE_SIZE.
>>
>> real    0m0.512s
>> user    0m0.042s
>> sys     0m0.165s
>>
>> Phil

Re: vchiq: Performance regression since 5.18-rc1

[Phil Elwell]

Hi Stefan,

On 23/05/2022 11:48, Stefan Wahren wrote:
> Hi Phil,
> 
> Am 23.05.22 um 11:29 schrieb Phil Elwell:
>> Hi Stefan,
>>
>> On 23/05/2022 07:19, Stefan Wahren wrote:
>>> Hi Paul,
>>>
>>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>>> Hi Paul,
>>>>>
>>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>>>
>>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>>>>
>>>>>>> Before commit:
>>>>>>>
>>>>>>> real    0m1,500s
>>>>>>> user    0m0,068s
>>>>>>> sys    0m0,846s
>>>>>>>
>>>>>>> After commit:
>>>>>>>
>>>>>>> real    7m11,449s
>>>>>>> user    0m2,049s
>>>>>>> sys    0m0,023s
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>>> Please feel free to try the patch shown below.  Or the pair of patches
>>>>>> from Rik here:
>>>>>>
>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>>>>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>>>>> minutes instead of ~ 1 second.
>>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>>> No, not explicit.
>>>>    That would
>>>> nullify my patch, but I would expect that Rik's patch would still provide
>>>> increased performance even in that case.
>>> I will retest with a fresh SD card image.
>>>>
>>>> Could you please characterize where the slowdown is occurring?
>>>
>>> Unfortunately i don't have a deep insight into driver and vchiq_test tool. 
>>> Just a user view.
>>>
>>> Do you think an strace would be a good starting point?
>>>
>>> @Phil Any advices to analyse this issue?
>>
>> Sending many small control packets:
>>
>>    vchiq_test -c 1 10000
>>
>> essentially tests interrupt latency. Using a small number of large bulk 
>> transfers:
>>
>>    vchiq_test -b 10000 1
>>
>> becomes a test of how long it takes to lock down pages. It also tests DMA 
>> transfer speeds, but since the DMA is run by the firmware (which you aren't 
>> changing), I think you can rule that.
> Thanks i will try.
>>
>> You may also find it helpful to include "force_turbo=1" in config.txt for more 
>> predictable results.
>>
>> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any 
>> performance problems:
> I assume you are using arm/bcm2709_defconfig and not arm/multi_v7_defconfig as me?

That's correct. Simply switching to multi_v7_defconfig breaks vchiq completely, 
presumably because it doesn't define CONFIG_BCM2835_VCHIQ.

Phil

>>
>> pi@raspberrypi:~$ time vchiq_test -f 1
>> Functional test - iters:1
>> ======== iteration 1 ========
>> Testing bulk transfer for alignment.
>> Testing bulk transfer at PAGE_SIZE.
>>
>> real    0m0.512s
>> user    0m0.042s
>> sys     0m0.165s
>>
>> Phil

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Phil,

Am 23.05.22 um 13:01 schrieb Phil Elwell:
> Hi Stefan,
>
> On 23/05/2022 11:48, Stefan Wahren wrote:
>> Hi Phil,
>>
>> Am 23.05.22 um 11:29 schrieb Phil Elwell:
>>> Hi Stefan,
>>>
>>> On 23/05/2022 07:19, Stefan Wahren wrote:
>>>> Hi Paul,
>>>>
>>>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>>>> Hi Paul,
>>>>>>
>>>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> while testing the staging/vc04_services/interface/vchiq_arm 
>>>>>>>> driver with my
>>>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge 
>>>>>>>> performance
>>>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>>>> lru_cache_disable: replace work queue synchronization with 
>>>>>>>> synchronize_rcu
>>>>>>>>
>>>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still 
>>>>>>>> working [1].
>>>>>>>>
>>>>>>>> Before commit:
>>>>>>>>
>>>>>>>> real    0m1,500s
>>>>>>>> user    0m0,068s
>>>>>>>> sys    0m0,846s
>>>>>>>>
>>>>>>>> After commit:
>>>>>>>>
>>>>>>>> real    7m11,449s
>>>>>>>> user    0m2,049s
>>>>>>>> sys    0m0,023s
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>>
>>>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>>>> Please feel free to try the patch shown below.  Or the pair of 
>>>>>>> patches
>>>>>>> from Rik here:
>>>>>>>
>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/ 
>>>>>>>
>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/ 
>>>>>>>
>>>>>> I tried your patch and Rik's patches but in both cases vchiq_test 
>>>>>> runs 7
>>>>>> minutes instead of ~ 1 second.
>>>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>>>> No, not explicit.
>>>>>    That would
>>>>> nullify my patch, but I would expect that Rik's patch would still 
>>>>> provide
>>>>> increased performance even in that case.
>>>> I will retest with a fresh SD card image.
>>>>>
>>>>> Could you please characterize where the slowdown is occurring?
>>>>
>>>> Unfortunately i don't have a deep insight into driver and 
>>>> vchiq_test tool. Just a user view.
>>>>
>>>> Do you think an strace would be a good starting point?
>>>>
>>>> @Phil Any advices to analyse this issue?
>>>
>>> Sending many small control packets:
>>>
>>>    vchiq_test -c 1 10000
>>>
>>> essentially tests interrupt latency. Using a small number of large 
>>> bulk transfers:
>>>
>>>    vchiq_test -b 10000 1
>>>
>>> becomes a test of how long it takes to lock down pages. It also 
>>> tests DMA transfer speeds, but since the DMA is run by the firmware 
>>> (which you aren't changing), I think you can rule that.
>> Thanks i will try.
>>>
>>> You may also find it helpful to include "force_turbo=1" in 
>>> config.txt for more predictable results.
>>>
>>> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not 
>>> seeing any performance problems:
>> I assume you are using arm/bcm2709_defconfig and not 
>> arm/multi_v7_defconfig as me?
>
> That's correct. Simply switching to multi_v7_defconfig breaks vchiq 
> completely, presumably because it doesn't define CONFIG_BCM2835_VCHIQ.
sorry, forgot to mention. I that i enable VCHIQ as module on top of 
multi_v7_defconfig.
>
> Phil
>
>>>
>>> pi@raspberrypi:~$ time vchiq_test -f 1
>>> Functional test - iters:1
>>> ======== iteration 1 ========
>>> Testing bulk transfer for alignment.
>>> Testing bulk transfer at PAGE_SIZE.
>>>
>>> real    0m0.512s
>>> user    0m0.042s
>>> sys     0m0.165s
>>>
>>> Phil

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Phil,

Am 23.05.22 um 13:01 schrieb Phil Elwell:
> Hi Stefan,
>
> On 23/05/2022 11:48, Stefan Wahren wrote:
>> Hi Phil,
>>
>> Am 23.05.22 um 11:29 schrieb Phil Elwell:
>>> Hi Stefan,
>>>
>>> On 23/05/2022 07:19, Stefan Wahren wrote:
>>>> Hi Paul,
>>>>
>>>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>>>> Hi Paul,
>>>>>>
>>>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> while testing the staging/vc04_services/interface/vchiq_arm 
>>>>>>>> driver with my
>>>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge 
>>>>>>>> performance
>>>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>>>> lru_cache_disable: replace work queue synchronization with 
>>>>>>>> synchronize_rcu
>>>>>>>>
>>>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still 
>>>>>>>> working [1].
>>>>>>>>
>>>>>>>> Before commit:
>>>>>>>>
>>>>>>>> real    0m1,500s
>>>>>>>> user    0m0,068s
>>>>>>>> sys    0m0,846s
>>>>>>>>
>>>>>>>> After commit:
>>>>>>>>
>>>>>>>> real    7m11,449s
>>>>>>>> user    0m2,049s
>>>>>>>> sys    0m0,023s
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>>
>>>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>>>> Please feel free to try the patch shown below.  Or the pair of 
>>>>>>> patches
>>>>>>> from Rik here:
>>>>>>>
>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/ 
>>>>>>>
>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/ 
>>>>>>>
>>>>>> I tried your patch and Rik's patches but in both cases vchiq_test 
>>>>>> runs 7
>>>>>> minutes instead of ~ 1 second.
>>>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>>>> No, not explicit.
>>>>>    That would
>>>>> nullify my patch, but I would expect that Rik's patch would still 
>>>>> provide
>>>>> increased performance even in that case.
>>>> I will retest with a fresh SD card image.
>>>>>
>>>>> Could you please characterize where the slowdown is occurring?
>>>>
>>>> Unfortunately i don't have a deep insight into driver and 
>>>> vchiq_test tool. Just a user view.
>>>>
>>>> Do you think an strace would be a good starting point?
>>>>
>>>> @Phil Any advices to analyse this issue?
>>>
>>> Sending many small control packets:
>>>
>>>    vchiq_test -c 1 10000
>>>
>>> essentially tests interrupt latency. Using a small number of large 
>>> bulk transfers:
>>>
>>>    vchiq_test -b 10000 1
>>>
>>> becomes a test of how long it takes to lock down pages. It also 
>>> tests DMA transfer speeds, but since the DMA is run by the firmware 
>>> (which you aren't changing), I think you can rule that.
>> Thanks i will try.
>>>
>>> You may also find it helpful to include "force_turbo=1" in 
>>> config.txt for more predictable results.
>>>
>>> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not 
>>> seeing any performance problems:
>> I assume you are using arm/bcm2709_defconfig and not 
>> arm/multi_v7_defconfig as me?
>
> That's correct. Simply switching to multi_v7_defconfig breaks vchiq 
> completely, presumably because it doesn't define CONFIG_BCM2835_VCHIQ.
sorry, forgot to mention. I that i enable VCHIQ as module on top of 
multi_v7_defconfig.
>
> Phil
>
>>>
>>> pi@raspberrypi:~$ time vchiq_test -f 1
>>> Functional test - iters:1
>>> ======== iteration 1 ========
>>> Testing bulk transfer for alignment.
>>> Testing bulk transfer at PAGE_SIZE.
>>>
>>> real    0m0.512s
>>> user    0m0.042s
>>> sys     0m0.165s
>>>
>>> Phil

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Phil Elwell]

On 23/05/2022 12:15, Stefan Wahren wrote:
> Hi Phil,
> 
> Am 23.05.22 um 13:01 schrieb Phil Elwell:
>> Hi Stefan,
>>
>> On 23/05/2022 11:48, Stefan Wahren wrote:
>>> Hi Phil,
>>>
>>> Am 23.05.22 um 11:29 schrieb Phil Elwell:
>>>> Hi Stefan,
>>>>
>>>> On 23/05/2022 07:19, Stefan Wahren wrote:
>>>>> Hi Paul,
>>>>>
>>>>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>>>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>>>>>
>>>>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>>>>>>
>>>>>>>>> Before commit:
>>>>>>>>>
>>>>>>>>> real    0m1,500s
>>>>>>>>> user    0m0,068s
>>>>>>>>> sys    0m0,846s
>>>>>>>>>
>>>>>>>>> After commit:
>>>>>>>>>
>>>>>>>>> real    7m11,449s
>>>>>>>>> user    0m2,049s
>>>>>>>>> sys    0m0,023s
>>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>>
>>>>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>>>>> Please feel free to try the patch shown below.  Or the pair of patches
>>>>>>>> from Rik here:
>>>>>>>>
>>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>>>>>>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>>>>>>> minutes instead of ~ 1 second.
>>>>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>>>>> No, not explicit.
>>>>>>    That would
>>>>>> nullify my patch, but I would expect that Rik's patch would still provide
>>>>>> increased performance even in that case.
>>>>> I will retest with a fresh SD card image.
>>>>>>
>>>>>> Could you please characterize where the slowdown is occurring?
>>>>>
>>>>> Unfortunately i don't have a deep insight into driver and vchiq_test tool. 
>>>>> Just a user view.
>>>>>
>>>>> Do you think an strace would be a good starting point?
>>>>>
>>>>> @Phil Any advices to analyse this issue?
>>>>
>>>> Sending many small control packets:
>>>>
>>>>    vchiq_test -c 1 10000
>>>>
>>>> essentially tests interrupt latency. Using a small number of large bulk 
>>>> transfers:
>>>>
>>>>    vchiq_test -b 10000 1
>>>>
>>>> becomes a test of how long it takes to lock down pages. It also tests DMA 
>>>> transfer speeds, but since the DMA is run by the firmware (which you aren't 
>>>> changing), I think you can rule that.
>>> Thanks i will try.
>>>>
>>>> You may also find it helpful to include "force_turbo=1" in config.txt for 
>>>> more predictable results.
>>>>
>>>> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any 
>>>> performance problems:
>>> I assume you are using arm/bcm2709_defconfig and not arm/multi_v7_defconfig 
>>> as me?
>>
>> That's correct. Simply switching to multi_v7_defconfig breaks vchiq 
>> completely, presumably because it doesn't define CONFIG_BCM2835_VCHIQ.
> sorry, forgot to mention. I that i enable VCHIQ as module on top of 
> multi_v7_defconfig.

Downstream tree with multi_v7_defconfig + CONFIG_BCM2835_VCHIQ:

pi@raspberrypi:~$ time vchiq_test -f 1
Functional test - iters:1
======== iteration 1 ========
Testing bulk transfer for alignment.
Testing bulk transfer at PAGE_SIZE.

real    0m0.566s
user    0m0.037s
sys     0m0.166s

Phil

Re: vchiq: Performance regression since 5.18-rc1

[Phil Elwell]

On 23/05/2022 12:15, Stefan Wahren wrote:
> Hi Phil,
> 
> Am 23.05.22 um 13:01 schrieb Phil Elwell:
>> Hi Stefan,
>>
>> On 23/05/2022 11:48, Stefan Wahren wrote:
>>> Hi Phil,
>>>
>>> Am 23.05.22 um 11:29 schrieb Phil Elwell:
>>>> Hi Stefan,
>>>>
>>>> On 23/05/2022 07:19, Stefan Wahren wrote:
>>>>> Hi Paul,
>>>>>
>>>>> Am 23.05.22 um 06:48 schrieb Paul E. McKenney:
>>>>>> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote:
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney:
>>>>>>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>>>>>
>>>>>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>>>>>>>
>>>>>>>>> Before commit:
>>>>>>>>>
>>>>>>>>> real    0m1,500s
>>>>>>>>> user    0m0,068s
>>>>>>>>> sys    0m0,846s
>>>>>>>>>
>>>>>>>>> After commit:
>>>>>>>>>
>>>>>>>>> real    7m11,449s
>>>>>>>>> user    0m2,049s
>>>>>>>>> sys    0m0,023s
>>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>>
>>>>>>>>> [1] - https://github.com/raspberrypi/userland
>>>>>>>> Please feel free to try the patch shown below.  Or the pair of patches
>>>>>>>> from Rik here:
>>>>>>>>
>>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/
>>>>>>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/
>>>>>>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7
>>>>>>> minutes instead of ~ 1 second.
>>>>>> That is surprising.  Do you boot with rcupdate.rcu_normal=1?
>>>>> No, not explicit.
>>>>>>    That would
>>>>>> nullify my patch, but I would expect that Rik's patch would still provide
>>>>>> increased performance even in that case.
>>>>> I will retest with a fresh SD card image.
>>>>>>
>>>>>> Could you please characterize where the slowdown is occurring?
>>>>>
>>>>> Unfortunately i don't have a deep insight into driver and vchiq_test tool. 
>>>>> Just a user view.
>>>>>
>>>>> Do you think an strace would be a good starting point?
>>>>>
>>>>> @Phil Any advices to analyse this issue?
>>>>
>>>> Sending many small control packets:
>>>>
>>>>    vchiq_test -c 1 10000
>>>>
>>>> essentially tests interrupt latency. Using a small number of large bulk 
>>>> transfers:
>>>>
>>>>    vchiq_test -b 10000 1
>>>>
>>>> becomes a test of how long it takes to lock down pages. It also tests DMA 
>>>> transfer speeds, but since the DMA is run by the firmware (which you aren't 
>>>> changing), I think you can rule that.
>>> Thanks i will try.
>>>>
>>>> You may also find it helpful to include "force_turbo=1" in config.txt for 
>>>> more predictable results.
>>>>
>>>> By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any 
>>>> performance problems:
>>> I assume you are using arm/bcm2709_defconfig and not arm/multi_v7_defconfig 
>>> as me?
>>
>> That's correct. Simply switching to multi_v7_defconfig breaks vchiq 
>> completely, presumably because it doesn't define CONFIG_BCM2835_VCHIQ.
> sorry, forgot to mention. I that i enable VCHIQ as module on top of 
> multi_v7_defconfig.

Downstream tree with multi_v7_defconfig + CONFIG_BCM2835_VCHIQ:

pi@raspberrypi:~$ time vchiq_test -f 1
Functional test - iters:1
======== iteration 1 ========
Testing bulk transfer for alignment.
Testing bulk transfer at PAGE_SIZE.

real    0m0.566s
user    0m0.037s
sys     0m0.166s

Phil

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Marcelo Tosatti]

On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
> > Hi,
> Hi,
> 
> > while testing the staging/vc04_services/interface/vchiq_arm driver with my
> > Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> > regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> > lru_cache_disable: replace work queue synchronization with synchronize_rcu
> > 
> > Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> 
> What about
> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
> 
> Sebastian

Stefan,

Can you please try the patch above ?

Re: vchiq: Performance regression since 5.18-rc1

[Marcelo Tosatti]

On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
> > Hi,
> Hi,
> 
> > while testing the staging/vc04_services/interface/vchiq_arm driver with my
> > Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> > regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> > lru_cache_disable: replace work queue synchronization with synchronize_rcu
> > 
> > Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> 
> What about
> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
> 
> Sebastian

Stefan,

Can you please try the patch above ?



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Marcelo,

Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>> Hi,
>> Hi,
>>
>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>
>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>> What about
>> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>
>> Sebastian
> Stefan,
>
> Can you please try the patch above ?

this was the same as Paul send. I think i need more time for 
investigation, maybe there is an issue with the application.

All i noticed so far is that in good case the CPU usage is around ~ 60 % 
and higher, while in bad case the CPU is almost idle. Also the issue is 
not reproducible with arm64/defconfig.

>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Marcelo,

Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>> Hi,
>> Hi,
>>
>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>
>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>> What about
>> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>
>> Sebastian
> Stefan,
>
> Can you please try the patch above ?

this was the same as Paul send. I think i need more time for 
investigation, maybe there is an issue with the application.

All i noticed so far is that in good case the CPU usage is around ~ 60 % 
and higher, while in bad case the CPU is almost idle. Also the issue is 
not reproducible with arm64/defconfig.

>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Sebastian Andrzej Siewior]

On 2022-05-25 16:07:47 [+0200], Stefan Wahren wrote:
> this was the same as Paul send. I think i need more time for investigation,
> maybe there is an issue with the application.

I haven't seen Paul referring to *that* patch. He pointed to some fs/
related changes.

Sebastian

Re: vchiq: Performance regression since 5.18-rc1

[Sebastian Andrzej Siewior]

On 2022-05-25 16:07:47 [+0200], Stefan Wahren wrote:
> this was the same as Paul send. I think i need more time for investigation,
> maybe there is an issue with the application.

I haven't seen Paul referring to *that* patch. He pointed to some fs/
related changes.

Sebastian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Paul E. McKenney]

On Wed, May 25, 2022 at 04:26:27PM +0200, Sebastian Andrzej Siewior wrote:
> On 2022-05-25 16:07:47 [+0200], Stefan Wahren wrote:
> > this was the same as Paul send. I think i need more time for investigation,
> > maybe there is an issue with the application.
> 
> I haven't seen Paul referring to *that* patch. He pointed to some fs/
> related changes.

True!  Both patches changed from a synchronize_rcu() to a
synchronize_rcu_expedited(), but different instances of synchronize_rcu().

							Thanx, Paul

Re: vchiq: Performance regression since 5.18-rc1

[Paul E. McKenney]

On Wed, May 25, 2022 at 04:26:27PM +0200, Sebastian Andrzej Siewior wrote:
> On 2022-05-25 16:07:47 [+0200], Stefan Wahren wrote:
> > this was the same as Paul send. I think i need more time for investigation,
> > maybe there is an issue with the application.
> 
> I haven't seen Paul referring to *that* patch. He pointed to some fs/
> related changes.

True!  Both patches changed from a synchronize_rcu() to a
synchronize_rcu_expedited(), but different instances of synchronize_rcu().

							Thanx, Paul

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Marcelo Tosatti]

On Wed, May 25, 2022 at 04:07:47PM +0200, Stefan Wahren wrote:
> Hi Marcelo,
> 
> Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
> > On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
> > > On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
> > > > Hi,
> > > Hi,
> > > 
> > > > while testing the staging/vc04_services/interface/vchiq_arm driver with my
> > > > Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> > > > regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> > > > lru_cache_disable: replace work queue synchronization with synchronize_rcu
> > > > 
> > > > Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> > > What about
> > > 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
> > > 
> > > Sebastian
> > Stefan,
> > 
> > Can you please try the patch above ?
> 
> this was the same as Paul send. I think i need more time for investigation,
> maybe there is an issue with the application.

To clarify: they are not the same patches.

> 
> All i noticed so far is that in good case the CPU usage is around ~ 60 % and
> higher, while in bad case the CPU is almost idle. Also the issue is not
> reproducible with arm64/defconfig.
> 
> > 
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> 

Re: vchiq: Performance regression since 5.18-rc1

[Marcelo Tosatti]

On Wed, May 25, 2022 at 04:07:47PM +0200, Stefan Wahren wrote:
> Hi Marcelo,
> 
> Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
> > On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
> > > On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
> > > > Hi,
> > > Hi,
> > > 
> > > > while testing the staging/vc04_services/interface/vchiq_arm driver with my
> > > > Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
> > > > regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
> > > > lru_cache_disable: replace work queue synchronization with synchronize_rcu
> > > > 
> > > > Usually i run "vchiq_test -f 1" to see the driver is still working [1].
> > > What about
> > > 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
> > > 
> > > Sebastian
> > Stefan,
> > 
> > Can you please try the patch above ?
> 
> this was the same as Paul send. I think i need more time for investigation,
> maybe there is an issue with the application.

To clarify: they are not the same patches.

> 
> All i noticed so far is that in good case the CPU usage is around ~ 60 % and
> higher, while in bad case the CPU is almost idle. Also the issue is not
> reproducible with arm64/defconfig.
> 
> > 
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Am 25.05.22 um 17:37 schrieb Marcelo Tosatti:
> On Wed, May 25, 2022 at 04:07:47PM +0200, Stefan Wahren wrote:
>> Hi Marcelo,
>>
>> Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
>>> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
>>>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>>>> Hi,
>>>> Hi,
>>>>
>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>
>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>> What about
>>>> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>>>
>>>> Sebastian
>>> Stefan,
>>>
>>> Can you please try the patch above ?
>> this was the same as Paul send. I think i need more time for investigation,
>> maybe there is an issue with the application.
> To clarify: they are not the same patches.
Thanks for pointing out. I will test it ASAP.
>
>> All i noticed so far is that in good case the CPU usage is around ~ 60 % and
>> higher, while in bad case the CPU is almost idle. Also the issue is not
>> reproducible with arm64/defconfig.
>>
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Am 25.05.22 um 17:37 schrieb Marcelo Tosatti:
> On Wed, May 25, 2022 at 04:07:47PM +0200, Stefan Wahren wrote:
>> Hi Marcelo,
>>
>> Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
>>> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
>>>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>>>> Hi,
>>>> Hi,
>>>>
>>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>>>
>>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>>>> What about
>>>> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>>>
>>>> Sebastian
>>> Stefan,
>>>
>>> Can you please try the patch above ?
>> this was the same as Paul send. I think i need more time for investigation,
>> maybe there is an issue with the application.
> To clarify: they are not the same patches.
Thanks for pointing out. I will test it ASAP.
>
>> All i noticed so far is that in good case the CPU usage is around ~ 60 % and
>> higher, while in bad case the CPU is almost idle. Also the issue is not
>> reproducible with arm64/defconfig.
>>
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Marcelo,
hi Sebastian,

Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>> Hi,
>> Hi,
>>
>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>
>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>> What about
>> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>
>> Sebastian
> Stefan,
>
> Can you please try the patch above ?

this patch fixes the regression. Great

Best regards

>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

Hi Marcelo,
hi Sebastian,

Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior wrote:
>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>> Hi,
>> Hi,
>>
>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my
>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
>>>
>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1].
>> What about
>> 	https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>
>> Sebastian
> Stefan,
>
> Can you please try the patch above ?

this patch fixes the regression. Great

Best regards

>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Stefan Wahren]

|||#regzbot fixed-by: 350e15124ee3|

Am 30.05.22 um 11:54 schrieb Stefan Wahren:
> Hi Marcelo,
> hi Sebastian,
>
> Am 25.05.22 um 15:56 schrieb Marcelo Tosatti:
>> On Mon, May 23, 2022 at 09:09:07AM +0200, Sebastian Andrzej Siewior 
>> wrote:
>>> On 2022-05-22 01:22:00 [+0200], Stefan Wahren wrote:
>>>> Hi,
>>> Hi,
>>>
>>>> while testing the staging/vc04_services/interface/vchiq_arm driver 
>>>> with my
>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>>>> lru_cache_disable: replace work queue synchronization with 
>>>> synchronize_rcu
>>>>
>>>> Usually i run "vchiq_test -f 1" to see the driver is still working 
>>>> [1].
>>> What about
>>>     https://lore.kernel.org/all/YmrWK%2FKoU1zrAxPI@fuller.cnet/
>>>
>>> Sebastian
>> Stefan,
>>
>> Can you please try the patch above ?
>
> this patch fixes the regression. Great
>
> Best regards
>
>>
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: vchiq: Performance regression since 5.18-rc1

[Thorsten Leemhuis]

On 23.05.22 11:28, Thorsten Leemhuis wrote:
> [TLDR: I'm adding this regression report to the list of tracked
> regressions; all text from me you find below is based on a few templates
> paragraphs you might have encountered already already in similar form.]
> 
> On 22.05.22 01:22, Stefan Wahren wrote:
>>
>> while testing the staging/vc04_services/interface/vchiq_arm driver with
>> my Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance
>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm:
>> lru_cache_disable: replace work queue synchronization with synchronize_rcu
> [...]
> #regzbot ^introduced ff042f4a9b050895a42cae893cc01fa2ca81b95
> #regzbot title mm: chiq_test runs 7 minutes instead of ~ 1 second.
> #regzbot ignore-activity

#regzbot fixed-by: 31733463372e8d