linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y
@ 2023-05-23  0:59 Wang Yugui
  2023-05-28 15:53 ` Wang Yugui
  0 siblings, 1 reply; 6+ messages in thread
From: Wang Yugui @ 2023-05-23  0:59 UTC (permalink / raw)
  To: cluster-devel
  Cc: linux-fsdevel, linux-xfs, Dave Chinner, Christoph Hellwig,
	Darrick J . Wong, Matthew Wilcox (Oracle)

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added  linux-xfs@ and linux-fsdevel@ because some related problem[1]
and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y becasue some related patches[2]
work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4@e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy@infradead.org/


test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
    gfs2 attr(T) GFS2_AF_ORLOV
   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
	-name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
	-name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
	-name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
	-name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.


5.15.y result
	fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
	fio  WRITE: bw=2599MiB/s (2725MB/s)

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2023/05/23



^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y
  2023-05-23  0:59 gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y Wang Yugui
@ 2023-05-28 15:53 ` Wang Yugui
  2023-07-10 13:19   ` [Cluster-devel] " Andreas Gruenbacher
  0 siblings, 1 reply; 6+ messages in thread
From: Wang Yugui @ 2023-05-28 15:53 UTC (permalink / raw)
  To: cluster-devel, linux-fsdevel, linux-xfs, Dave Chinner,
	Christoph Hellwig, Darrick J . Wong, Matthew Wilcox (Oracle)

Hi,

> Hi,
> 
> gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.
> 
> we added  linux-xfs@ and linux-fsdevel@ because some related problem[1]
> and related patches[2].
> 
> we compared 6.4-rc3(rather than 6.1.y) to 5.15.y becasue some related patches[2]
> work only for 6.4 now.
> 
> [1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4@e16-tech.com/
> [2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy@infradead.org/
> 
> 
> test case:
> 1) PCIe3 SSD *4 with LVM
> 2) gfs2 lock_nolock
>     gfs2 attr(T) GFS2_AF_ORLOV
>    # chattr +T /mnt/test
> 3) fio
> fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
> -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
> 	-name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
> 	-name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
> 	-name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
> 	-name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
> 4) patches[2] are applied to 6.4-rc3.
> 
> 
> 5.15.y result
> 	fio WRITE: bw=5139MiB/s (5389MB/s),
> 6.4-rc3 result
> 	fio  WRITE: bw=2599MiB/s (2725MB/s)

more test result:

5.17.0	WRITE: bw=4988MiB/s (5231MB/s)
5.18.0	WRITE: bw=5165MiB/s (5416MB/s)
5.19.0	WRITE: bw=5511MiB/s (5779MB/s)
6.0.5	WRITE: bw=3055MiB/s (3203MB/s),	WRITE: bw=3225MiB/s (3382MB/s)
6.1.30	WRITE: bw=2579MiB/s (2705MB/s)

so this regression  happen in some code introduced in 6.0,
and maybe some minor regression in 6.1 too?

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2023/05/28


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Cluster-devel] gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y
  2023-05-28 15:53 ` Wang Yugui
@ 2023-07-10 13:19   ` Andreas Gruenbacher
  2023-07-11  0:58     ` Wang Yugui
  2023-07-11  2:08     ` Matthew Wilcox
  0 siblings, 2 replies; 6+ messages in thread
From: Andreas Gruenbacher @ 2023-07-10 13:19 UTC (permalink / raw)
  To: Wang Yugui
  Cc: Bob Peterson, cluster-devel, linux-fsdevel, linux-xfs,
	Dave Chinner, Christoph Hellwig, Darrick J . Wong,
	Matthew Wilcox (Oracle)

Hi Wang Yugui,

On Sun, May 28, 2023 at 5:53 PM Wang Yugui <wangyugui@e16-tech.com> wrote:
> Hi,
>
> > Hi,
> >
> > gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.
> >
> > we added  linux-xfs@ and linux-fsdevel@ because some related problem[1]
> > and related patches[2].
> >
> > we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2]
> > work only for 6.4 now.
> >
> > [1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4@e16-tech.com/
> > [2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy@infradead.org/
> >
> >
> > test case:
> > 1) PCIe3 SSD *4 with LVM
> > 2) gfs2 lock_nolock
> >     gfs2 attr(T) GFS2_AF_ORLOV
> >    # chattr +T /mnt/test
> > 3) fio
> > fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
> > -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
> >       -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
> >       -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
> >       -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
> >       -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
> > 4) patches[2] are applied to 6.4-rc3.
> >
> >
> > 5.15.y result
> >       fio WRITE: bw=5139MiB/s (5389MB/s),
> > 6.4-rc3 result
> >       fio  WRITE: bw=2599MiB/s (2725MB/s)
>
> more test result:
>
> 5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
> 5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
> 5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
> 6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
> 6.1.30  WRITE: bw=2579MiB/s (2705MB/s)
>
> so this regression  happen in some code introduced in 6.0,
> and maybe some minor regression in 6.1 too?

thanks for this bug report. Bob has noticed a similar looking
performance regression recently, and it turned out that commit
e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now")
inadvertently caused buffered writes to fall back to writing single
pages instead of multiple pages at once. That patch was added in
v5.18, so it doesn't perfectly align with the regression history
you're reporting, but maybe there's something else going on that we're
not aware of.

In any case, the regression introduced by commit e1fa9ea85ce8 should
be fixed by commit c8ed1b359312 ("gfs2: Fix duplicate
should_fault_in_pages() call"), which ended up in v6.5-rc1.

Could you please check where we end up with that fix?

Thank you very much,
Andreas


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Cluster-devel] gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y
  2023-07-10 13:19   ` [Cluster-devel] " Andreas Gruenbacher
@ 2023-07-11  0:58     ` Wang Yugui
  2023-07-11  1:58       ` Wang Yugui
  2023-07-11  2:08     ` Matthew Wilcox
  1 sibling, 1 reply; 6+ messages in thread
From: Wang Yugui @ 2023-07-11  0:58 UTC (permalink / raw)
  To: Andreas Gruenbacher
  Cc: Bob Peterson, cluster-devel, linux-fsdevel, linux-xfs,
	Dave Chinner, Christoph Hellwig, Darrick J . Wong,
	Matthew Wilcox (Oracle)

Hi,

> Hi Wang Yugui,
> 
> On Sun, May 28, 2023 at 5:53?PM Wang Yugui <wangyugui@e16-tech.com> wrote:
> > Hi,
> >
> > > Hi,
> > >
> > > gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.
> > >
> > > we added  linux-xfs@ and linux-fsdevel@ because some related problem[1]
> > > and related patches[2].
> > >
> > > we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2]
> > > work only for 6.4 now.
> > >
> > > [1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4@e16-tech.com/
> > > [2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy@infradead.org/
> > >
> > >
> > > test case:
> > > 1) PCIe3 SSD *4 with LVM
> > > 2) gfs2 lock_nolock
> > >     gfs2 attr(T) GFS2_AF_ORLOV
> > >    # chattr +T /mnt/test
> > > 3) fio
> > > fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
> > > -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
> > >       -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
> > >       -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
> > >       -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
> > >       -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
> > > 4) patches[2] are applied to 6.4-rc3.
> > >
> > >
> > > 5.15.y result
> > >       fio WRITE: bw=5139MiB/s (5389MB/s),
> > > 6.4-rc3 result
> > >       fio  WRITE: bw=2599MiB/s (2725MB/s)
> >
> > more test result:
> >
> > 5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
> > 5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
> > 5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
> > 6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
> > 6.1.30  WRITE: bw=2579MiB/s (2705MB/s)
> >
> > so this regression  happen in some code introduced in 6.0,
> > and maybe some minor regression in 6.1 too?
> 
> thanks for this bug report. Bob has noticed a similar looking
> performance regression recently, and it turned out that commit
> e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now")
> inadvertently caused buffered writes to fall back to writing single
> pages instead of multiple pages at once. That patch was added in
> v5.18, so it doesn't perfectly align with the regression history
> you're reporting, but maybe there's something else going on that we're
> not aware of.
> 
> In any case, the regression introduced by commit e1fa9ea85ce8 should
> be fixed by commit c8ed1b359312 ("gfs2: Fix duplicate
> should_fault_in_pages() call"), which ended up in v6.5-rc1.
> 
> Could you please check where we end up with that fix?

I applied c8ed1b359312 on 6.1.36.
# the build/test of 6.5-rc1 is yet not ready.

fio performance result:
  WRITE: bw=2683MiB/s (2813MB/s)

but  the performance of fio 'Laying out IO file' is improved.
Jobs: 4 (f=4): [F(4)][100.0%][w=5168MiB/s][w=5168 IOPS][eta 00m:00s]

so there seems 2 problems,  one is fixed by c8ed1b359312.
but another is still left.

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2023/07/11



^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Cluster-devel] gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y
  2023-07-11  0:58     ` Wang Yugui
@ 2023-07-11  1:58       ` Wang Yugui
  0 siblings, 0 replies; 6+ messages in thread
From: Wang Yugui @ 2023-07-11  1:58 UTC (permalink / raw)
  To: Andreas Gruenbacher
  Cc: Bob Peterson, cluster-devel, linux-fsdevel, linux-xfs,
	Dave Chinner, Christoph Hellwig, Darrick J . Wong,
	Matthew Wilcox (Oracle)

Hi,

> Hi,
> 
> > Hi Wang Yugui,
> > 
> > On Sun, May 28, 2023 at 5:53?PM Wang Yugui <wangyugui@e16-tech.com> wrote:
> > > Hi,
> > >
> > > > Hi,
> > > >
> > > > gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.
> > > >
> > > > we added  linux-xfs@ and linux-fsdevel@ because some related problem[1]
> > > > and related patches[2].
> > > >
> > > > we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2]
> > > > work only for 6.4 now.
> > > >
> > > > [1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4@e16-tech.com/
> > > > [2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy@infradead.org/
> > > >
> > > >
> > > > test case:
> > > > 1) PCIe3 SSD *4 with LVM
> > > > 2) gfs2 lock_nolock
> > > >     gfs2 attr(T) GFS2_AF_ORLOV
> > > >    # chattr +T /mnt/test
> > > > 3) fio
> > > > fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
> > > > -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
> > > >       -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
> > > >       -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
> > > >       -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
> > > >       -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
> > > > 4) patches[2] are applied to 6.4-rc3.
> > > >
> > > >
> > > > 5.15.y result
> > > >       fio WRITE: bw=5139MiB/s (5389MB/s),
> > > > 6.4-rc3 result
> > > >       fio  WRITE: bw=2599MiB/s (2725MB/s)
> > >
> > > more test result:
> > >
> > > 5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
> > > 5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
> > > 5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
> > > 6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
> > > 6.1.30  WRITE: bw=2579MiB/s (2705MB/s)
> > >
> > > so this regression  happen in some code introduced in 6.0,
> > > and maybe some minor regression in 6.1 too?
> > 
> > thanks for this bug report. Bob has noticed a similar looking
> > performance regression recently, and it turned out that commit
> > e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now")
> > inadvertently caused buffered writes to fall back to writing single
> > pages instead of multiple pages at once. That patch was added in
> > v5.18, so it doesn't perfectly align with the regression history
> > you're reporting, but maybe there's something else going on that we're
> > not aware of.
> > 
> > In any case, the regression introduced by commit e1fa9ea85ce8 should
> > be fixed by commit c8ed1b359312 ("gfs2: Fix duplicate
> > should_fault_in_pages() call"), which ended up in v6.5-rc1.
> > 
> > Could you please check where we end up with that fix?
> 
> I applied c8ed1b359312 on 6.1.36.
> # the build/test of 6.5-rc1 is yet not ready.
> 
> fio performance result:
>   WRITE: bw=2683MiB/s (2813MB/s)
> 
> but  the performance of fio 'Laying out IO file' is improved.
> Jobs: 4 (f=4): [F(4)][100.0%][w=5168MiB/s][w=5168 IOPS][eta 00m:00s]
> 
> so there seems 2 problems,  one is fixed by c8ed1b359312.
> but another is still left.


We build/test on 6.5-rc1 too.
fio  WRITE: bw=2643MiB/s (2771MB/s)

the performance of fio 'Laying out IO file'.
Jobs: 4 (f=4): [F(4)][100.0%][w=4884MiB/s][w=4884 IOPS][eta 00m:00s]

so the performance result on 6.5-rc1 is the same level as
6.1.36 with c8ed1b359312.

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2023/07/11



^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Cluster-devel] gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y
  2023-07-10 13:19   ` [Cluster-devel] " Andreas Gruenbacher
  2023-07-11  0:58     ` Wang Yugui
@ 2023-07-11  2:08     ` Matthew Wilcox
  1 sibling, 0 replies; 6+ messages in thread
From: Matthew Wilcox @ 2023-07-11  2:08 UTC (permalink / raw)
  To: Andreas Gruenbacher
  Cc: Wang Yugui, Bob Peterson, cluster-devel, linux-fsdevel,
	linux-xfs, Dave Chinner, Christoph Hellwig, Darrick J . Wong

On Mon, Jul 10, 2023 at 03:19:54PM +0200, Andreas Gruenbacher wrote:
> Hi Wang Yugui,
> 
> On Sun, May 28, 2023 at 5:53 PM Wang Yugui <wangyugui@e16-tech.com> wrote:
> > Hi,
> >
> > > Hi,
> > >
> > > gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.
> > >
> > > we added  linux-xfs@ and linux-fsdevel@ because some related problem[1]
> > > and related patches[2].
> > >
> > > we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2]
> > > work only for 6.4 now.
> > >
> > > [1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4@e16-tech.com/
> > > [2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy@infradead.org/
> > >
> > >
> > > test case:
> > > 1) PCIe3 SSD *4 with LVM
> > > 2) gfs2 lock_nolock
> > >     gfs2 attr(T) GFS2_AF_ORLOV
> > >    # chattr +T /mnt/test
> > > 3) fio
> > > fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
> > > -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
> > >       -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
> > >       -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
> > >       -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
> > >       -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
> > > 4) patches[2] are applied to 6.4-rc3.
> > >
> > >
> > > 5.15.y result
> > >       fio WRITE: bw=5139MiB/s (5389MB/s),
> > > 6.4-rc3 result
> > >       fio  WRITE: bw=2599MiB/s (2725MB/s)
> >
> > more test result:
> >
> > 5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
> > 5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
> > 5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
> > 6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
> > 6.1.30  WRITE: bw=2579MiB/s (2705MB/s)
> >
> > so this regression  happen in some code introduced in 6.0,
> > and maybe some minor regression in 6.1 too?
> 
> thanks for this bug report. Bob has noticed a similar looking
> performance regression recently, and it turned out that commit
> e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now")
> inadvertently caused buffered writes to fall back to writing single
> pages instead of multiple pages at once. That patch was added in
> v5.18, so it doesn't perfectly align with the regression history
> you're reporting, but maybe there's something else going on that we're
> not aware of.

Dave gave a good explanation of the problem here:

https://lore.kernel.org/linux-xfs/ZKybxCxzmuI1TFYn@dread.disaster.area/

It's a pagecache locking contention problem rather than an individual
filesystem problem.

... are you interested in supporting large folios in gfs2?  ;-)

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-07-11  2:09 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-23  0:59 gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y Wang Yugui
2023-05-28 15:53 ` Wang Yugui
2023-07-10 13:19   ` [Cluster-devel] " Andreas Gruenbacher
2023-07-11  0:58     ` Wang Yugui
2023-07-11  1:58       ` Wang Yugui
2023-07-11  2:08     ` Matthew Wilcox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).