Re: disable large folios for shmem file used by xfs xfile

Re: disable large folios for shmem file used by xfs xfile

[Matthew Wilcox]

On Wed, Jan 10, 2024 at 10:21:07AM +0100, Christoph Hellwig wrote:
> Hi all,
> 
> Darrick reported that the fairly new XFS xfile code blows up when force
> enabling large folio for shmem.  This series fixes this quickly by disabling
> large folios for this particular shmem file for now until it can be fixed
> properly, which will be a lot more invasive.
> 
> I've added most of you to the CC list as I suspect most other users of
> shmem_file_setup and friends will have similar issues.

The graphics users _want_ to use large folios.  I'm pretty sure they've
been tested with this.  It's just XFS that didn't know about this
feature of shmem.

Re: disable large folios for shmem file used by xfs xfile

[Joonas Lahtinen]

Quoting Matthew Wilcox (2024-01-10 14:37:18)
> On Wed, Jan 10, 2024 at 10:21:07AM +0100, Christoph Hellwig wrote:
> > Hi all,
> > 
> > Darrick reported that the fairly new XFS xfile code blows up when force
> > enabling large folio for shmem.  This series fixes this quickly by disabling
> > large folios for this particular shmem file for now until it can be fixed
> > properly, which will be a lot more invasive.
> > 
> > I've added most of you to the CC list as I suspect most other users of
> > shmem_file_setup and friends will have similar issues.
> 
> The graphics users _want_ to use large folios.  I'm pretty sure they've
> been tested with this.

Correct. We've done quite a bit of optimization in userspace and
enabling in kernel to take advantage of page sizes of 2M and beyond.

However we specifically pass "huge=within_size" to vfs_kern_mount when
creating a private mount of tmpfs for the purpose of i915 created
allocations.

Older hardware also had some address hashing bugs where 2M aligned
memory caused a lot of collisions in TLB so we don't enable it always.

You can see drivers/gpu/drm/i915/gem/i915_gemfs.c function
i915_gemfs_init for details and references.

So in short, functionality wise we should be fine either default
for using 2M pages or not. If they become the default, we'd probably
want an option that would still be able to prevent them for performance
regression reasons on older hardware.

Regards, Joonas

> It's just XFS that didn't know about this
> feature of shmem.

Re: disable large folios for shmem file used by xfs xfile

[Joonas Lahtinen]

Quoting Joonas Lahtinen (2024-01-10 17:20:24)
> Quoting Matthew Wilcox (2024-01-10 14:37:18)
> > On Wed, Jan 10, 2024 at 10:21:07AM +0100, Christoph Hellwig wrote:
> > > Hi all,
> > > 
> > > Darrick reported that the fairly new XFS xfile code blows up when force
> > > enabling large folio for shmem.  This series fixes this quickly by disabling
> > > large folios for this particular shmem file for now until it can be fixed
> > > properly, which will be a lot more invasive.
> > > 
> > > I've added most of you to the CC list as I suspect most other users of
> > > shmem_file_setup and friends will have similar issues.
> > 
> > The graphics users _want_ to use large folios.  I'm pretty sure they've
> > been tested with this.
> 
> Correct. We've done quite a bit of optimization in userspace and
> enabling in kernel to take advantage of page sizes of 2M and beyond.
> 
> However we specifically pass "huge=within_size" to vfs_kern_mount when
> creating a private mount of tmpfs for the purpose of i915 created
> allocations.
> 
> Older hardware also had some address hashing bugs where 2M aligned
> memory caused a lot of collisions in TLB so we don't enable it always.
> 
> You can see drivers/gpu/drm/i915/gem/i915_gemfs.c function
> i915_gemfs_init for details and references.
> 
> So in short, functionality wise we should be fine either default
> for using 2M pages or not. If they become the default, we'd probably
> want an option that would still be able to prevent them for performance
> regression reasons on older hardware.

To maybe write out my concern better:

Is there plan to enable huge pages by default in shmem?

If not I guess we should be pretty good with the way current code is, force
enabling them just might bring out some performance, so we might want to add
a warning for that.

If there is, then we'll probably want to in sync with those default changes
apply a similar call to block them on older HW.

Regards, Joonas

> 
> Regards, Joonas
> 
> > It's just XFS that didn't know about this
> > feature of shmem.

Re: disable large folios for shmem file used by xfs xfile

[Matthew Wilcox]

On Wed, Jan 10, 2024 at 05:28:22PM +0200, Joonas Lahtinen wrote:
> Quoting Joonas Lahtinen (2024-01-10 17:20:24)
> > However we specifically pass "huge=within_size" to vfs_kern_mount when
> > creating a private mount of tmpfs for the purpose of i915 created
> > allocations.
> > 
> > Older hardware also had some address hashing bugs where 2M aligned
> > memory caused a lot of collisions in TLB so we don't enable it always.
> > 
> > You can see drivers/gpu/drm/i915/gem/i915_gemfs.c function
> > i915_gemfs_init for details and references.
> > 
> > So in short, functionality wise we should be fine either default
> > for using 2M pages or not. If they become the default, we'd probably
> > want an option that would still be able to prevent them for performance
> > regression reasons on older hardware.
> 
> To maybe write out my concern better:
> 
> Is there plan to enable huge pages by default in shmem?

Not in the next kernel release, but eventually the plan is to allow
arbitrary order folios to be used in shmem.  So you could ask it to create
a 256kB folio for you, if that's the right size to manage memory in.

How shmem and its various users go about choosing the right size is not
quite clear to me yet.  Perhaps somebody else will do it before I get
to it; I have a lot of different sub-projects to work on at the moment,
and shmem isn't blocking any of them.  And I have a sneaking suspicion
that more work is needed in the swap code to deal with arbitrary order
folios, so that's another reason for me to delay looking at this ;-)

Re: disable large folios for shmem file used by xfs xfile

[Andrew Morton]

On Wed, 10 Jan 2024 10:21:07 +0100 Christoph Hellwig <hch@lst.de> wrote:

> Hi all,
> 
> Darrick reported that the fairly new XFS xfile code blows up when force
> enabling large folio for shmem.  This series fixes this quickly by disabling
> large folios for this particular shmem file for now until it can be fixed
> properly, which will be a lot more invasive.
> 

Patches seems reasonable as a short-term only-affects-xfs thing.

I assume that kernels which contain 137db333b29186 ("xfs: teach xfile
to pass back direct-map pages to caller") want this, so a Fixes: that
and a cc:stable are appropriate?

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Darrick J. Wong]

On Wed, Jan 10, 2024 at 10:21:09AM +0100, Christoph Hellwig wrote:
> The xfarray code will crash if large folios are force enabled using:
> 
>    echo force > /sys/kernel/mm/transparent_hugepage/shmem_enabled
> 
> Fixing this will require a bit of an API change, and prefeably sorting out
> the hwpoison story for pages vs folio and where it is placed in the shmem
> API.  For now use this one liner to disable large folios.
> 
> Reported-by: Darrick J. Wong <djwong@kernel.org>
> Signed-off-by: Christoph Hellwig <hch@lst.de>

Can someone who knows more about shmem.c than I do please review
https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
so that I can feel slightly more confident as hch and I sort through the
xfile.c issues?

For this patch,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  fs/xfs/scrub/xfile.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/fs/xfs/scrub/xfile.c b/fs/xfs/scrub/xfile.c
> index 090c3ead43fdf1..1a8d1bedd0b0dc 100644
> --- a/fs/xfs/scrub/xfile.c
> +++ b/fs/xfs/scrub/xfile.c
> @@ -94,6 +94,11 @@ xfile_create(
>  
>  	lockdep_set_class(&inode->i_rwsem, &xfile_i_mutex_key);
>  
> +	/*
> +	 * We're not quite ready for large folios yet.
> +	 */
> +	mapping_clear_large_folios(inode->i_mapping);
> +
>  	trace_xfile_create(xf);
>  
>  	*xfilep = xf;
> -- 
> 2.39.2
> 
> 

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Darrick J. Wong]

On Wed, Jan 10, 2024 at 09:55:15AM -0800, Darrick J. Wong wrote:
> On Wed, Jan 10, 2024 at 10:21:09AM +0100, Christoph Hellwig wrote:
> > The xfarray code will crash if large folios are force enabled using:
> > 
> >    echo force > /sys/kernel/mm/transparent_hugepage/shmem_enabled
> > 
> > Fixing this will require a bit of an API change, and prefeably sorting out
> > the hwpoison story for pages vs folio and where it is placed in the shmem
> > API.  For now use this one liner to disable large folios.
> > 
> > Reported-by: Darrick J. Wong <djwong@kernel.org>
> > Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> Can someone who knows more about shmem.c than I do please review
> https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
> so that I can feel slightly more confident as hch and I sort through the
> xfile.c issues?
> 
> For this patch,
> Reviewed-by: Darrick J. Wong <djwong@kernel.org>

...except that I'm still getting 2M THPs even with this enabled, so I
guess either we get to fix it now, or create our own private tmpfs mount
so that we can pass in huge=never, similar to what i915 does. :(

--D

> --D
> 
> > ---
> >  fs/xfs/scrub/xfile.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/fs/xfs/scrub/xfile.c b/fs/xfs/scrub/xfile.c
> > index 090c3ead43fdf1..1a8d1bedd0b0dc 100644
> > --- a/fs/xfs/scrub/xfile.c
> > +++ b/fs/xfs/scrub/xfile.c
> > @@ -94,6 +94,11 @@ xfile_create(
> >  
> >  	lockdep_set_class(&inode->i_rwsem, &xfile_i_mutex_key);
> >  
> > +	/*
> > +	 * We're not quite ready for large folios yet.
> > +	 */
> > +	mapping_clear_large_folios(inode->i_mapping);
> > +
> >  	trace_xfile_create(xf);
> >  
> >  	*xfilep = xf;
> > -- 
> > 2.39.2
> > 
> > 
> 

Re: disable large folios for shmem file used by xfs xfile

[Daniel Gomez]

On Wed, Jan 10, 2024 at 4:35 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Wed, Jan 10, 2024 at 05:28:22PM +0200, Joonas Lahtinen wrote:
> > Quoting Joonas Lahtinen (2024-01-10 17:20:24)
> > > However we specifically pass "huge=within_size" to vfs_kern_mount when
> > > creating a private mount of tmpfs for the purpose of i915 created
> > > allocations.
> > >
> > > Older hardware also had some address hashing bugs where 2M aligned
> > > memory caused a lot of collisions in TLB so we don't enable it always.
> > >
> > > You can see drivers/gpu/drm/i915/gem/i915_gemfs.c function
> > > i915_gemfs_init for details and references.
> > >
> > > So in short, functionality wise we should be fine either default
> > > for using 2M pages or not. If they become the default, we'd probably
> > > want an option that would still be able to prevent them for performance
> > > regression reasons on older hardware.
> >
> > To maybe write out my concern better:
> >
> > Is there plan to enable huge pages by default in shmem?
>
> Not in the next kernel release, but eventually the plan is to allow
> arbitrary order folios to be used in shmem.  So you could ask it to create
> a 256kB folio for you, if that's the right size to manage memory in.
>
> How shmem and its various users go about choosing the right size is not
> quite clear to me yet.  Perhaps somebody else will do it before I get
> to it; I have a lot of different sub-projects to work on at the moment,
> and shmem isn't blocking any of them.  And I have a sneaking suspicion
> that more work is needed in the swap code to deal with arbitrary order
> folios, so that's another reason for me to delay looking at this ;-)

I have sent large folios support for shmem for the write and fallocate
path some releases ago. The main problem I was facing was a current
upstream problem with huge pages when seeking holes/data (fstests
generic/285 and generic/436). The strategy suggested was to use large
folios in an opportunistic way based on the file size. This hit the
same problem we currently have with huge pages and I considered that a
regression. We have made some progress to fix seeking in huge pages
upstream but is not yet finished. I can send the patches tomorrow for
further discussion.

>

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Andrew Morton]

On Wed, 10 Jan 2024 12:04:51 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:

> > > Fixing this will require a bit of an API change, and prefeably sorting out
> > > the hwpoison story for pages vs folio and where it is placed in the shmem
> > > API.  For now use this one liner to disable large folios.
> > > 
> > > Reported-by: Darrick J. Wong <djwong@kernel.org>
> > > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > 
> > Can someone who knows more about shmem.c than I do please review
> > https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
> > so that I can feel slightly more confident as hch and I sort through the
> > xfile.c issues?
> > 
> > For this patch,
> > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> 
> ...except that I'm still getting 2M THPs even with this enabled, so I
> guess either we get to fix it now, or create our own private tmpfs mount
> so that we can pass in huge=never, similar to what i915 does. :(

What is "this"?  Are you saying that $Subject doesn't work, or that the
above-linked please-review patch doesn't work?

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Matthew Wilcox]

On Thu, Jan 11, 2024 at 02:00:53PM -0800, Andrew Morton wrote:
> On Wed, 10 Jan 2024 12:04:51 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:
> 
> > > > Fixing this will require a bit of an API change, and prefeably sorting out
> > > > the hwpoison story for pages vs folio and where it is placed in the shmem
> > > > API.  For now use this one liner to disable large folios.
> > > > 
> > > > Reported-by: Darrick J. Wong <djwong@kernel.org>
> > > > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > > 
> > > Can someone who knows more about shmem.c than I do please review
> > > https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
> > > so that I can feel slightly more confident as hch and I sort through the
> > > xfile.c issues?
> > > 
> > > For this patch,
> > > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> > 
> > ...except that I'm still getting 2M THPs even with this enabled, so I
> > guess either we get to fix it now, or create our own private tmpfs mount
> > so that we can pass in huge=never, similar to what i915 does. :(
> 
> What is "this"?  Are you saying that $Subject doesn't work, or that the
> above-linked please-review patch doesn't work?

shmem pays no attention to the mapping_large_folio_support() flag,
so the proposed fix doesn't work.  It ought to, but it has its own way
of doing it that predates mapping_large_folio_support existing.

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Darrick J. Wong]

On Thu, Jan 11, 2024 at 10:45:53PM +0000, Matthew Wilcox wrote:
> On Thu, Jan 11, 2024 at 02:00:53PM -0800, Andrew Morton wrote:
> > On Wed, 10 Jan 2024 12:04:51 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:
> > 
> > > > > Fixing this will require a bit of an API change, and prefeably sorting out
> > > > > the hwpoison story for pages vs folio and where it is placed in the shmem
> > > > > API.  For now use this one liner to disable large folios.
> > > > > 
> > > > > Reported-by: Darrick J. Wong <djwong@kernel.org>
> > > > > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > > > 
> > > > Can someone who knows more about shmem.c than I do please review
> > > > https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
> > > > so that I can feel slightly more confident as hch and I sort through the
> > > > xfile.c issues?
> > > > 
> > > > For this patch,
> > > > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> > > 
> > > ...except that I'm still getting 2M THPs even with this enabled, so I
> > > guess either we get to fix it now, or create our own private tmpfs mount
> > > so that we can pass in huge=never, similar to what i915 does. :(
> > 
> > What is "this"?  Are you saying that $Subject doesn't work, or that the
> > above-linked please-review patch doesn't work?
> 
> shmem pays no attention to the mapping_large_folio_support() flag,
> so the proposed fix doesn't work.  It ought to, but it has its own way
> of doing it that predates mapping_large_folio_support existing.

Yep.  It turned out to be easier to fix xfile.c to deal with large
folios than I thought it would be.  Or so I think.  We'll see what
happens on fstestscloud overnight.

--D

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Andrew Morton]

On Thu, 11 Jan 2024 18:22:50 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:

> On Thu, Jan 11, 2024 at 10:45:53PM +0000, Matthew Wilcox wrote:
> > On Thu, Jan 11, 2024 at 02:00:53PM -0800, Andrew Morton wrote:
> > > On Wed, 10 Jan 2024 12:04:51 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:
> > > 
> > > > > > Fixing this will require a bit of an API change, and prefeably sorting out
> > > > > > the hwpoison story for pages vs folio and where it is placed in the shmem
> > > > > > API.  For now use this one liner to disable large folios.
> > > > > > 
> > > > > > Reported-by: Darrick J. Wong <djwong@kernel.org>
> > > > > > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > > > > 
> > > > > Can someone who knows more about shmem.c than I do please review
> > > > > https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
> > > > > so that I can feel slightly more confident as hch and I sort through the
> > > > > xfile.c issues?
> > > > > 
> > > > > For this patch,
> > > > > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> > > > 
> > > > ...except that I'm still getting 2M THPs even with this enabled, so I
> > > > guess either we get to fix it now, or create our own private tmpfs mount
> > > > so that we can pass in huge=never, similar to what i915 does. :(
> > > 
> > > What is "this"?  Are you saying that $Subject doesn't work, or that the
> > > above-linked please-review patch doesn't work?
> > 
> > shmem pays no attention to the mapping_large_folio_support() flag,
> > so the proposed fix doesn't work.  It ought to, but it has its own way
> > of doing it that predates mapping_large_folio_support existing.
> 
> Yep.  It turned out to be easier to fix xfile.c to deal with large
> folios than I thought it would be.  Or so I think.  We'll see what
> happens on fstestscloud overnight.

Where do we stand with this?  Should I merge these two patches into
6.8-rcX, cc:stable?

Re: [PATCH 2/2] xfs: disable large folio support in xfile_create

[Darrick J. Wong]

On Wed, Feb 07, 2024 at 05:56:21PM -0800, Andrew Morton wrote:
> On Thu, 11 Jan 2024 18:22:50 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:
> 
> > On Thu, Jan 11, 2024 at 10:45:53PM +0000, Matthew Wilcox wrote:
> > > On Thu, Jan 11, 2024 at 02:00:53PM -0800, Andrew Morton wrote:
> > > > On Wed, 10 Jan 2024 12:04:51 -0800 "Darrick J. Wong" <djwong@kernel.org> wrote:
> > > > 
> > > > > > > Fixing this will require a bit of an API change, and prefeably sorting out
> > > > > > > the hwpoison story for pages vs folio and where it is placed in the shmem
> > > > > > > API.  For now use this one liner to disable large folios.
> > > > > > > 
> > > > > > > Reported-by: Darrick J. Wong <djwong@kernel.org>
> > > > > > > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > > > > > 
> > > > > > Can someone who knows more about shmem.c than I do please review
> > > > > > https://lore.kernel.org/linux-xfs/20240103084126.513354-4-hch@lst.de/
> > > > > > so that I can feel slightly more confident as hch and I sort through the
> > > > > > xfile.c issues?
> > > > > > 
> > > > > > For this patch,
> > > > > > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> > > > > 
> > > > > ...except that I'm still getting 2M THPs even with this enabled, so I
> > > > > guess either we get to fix it now, or create our own private tmpfs mount
> > > > > so that we can pass in huge=never, similar to what i915 does. :(
> > > > 
> > > > What is "this"?  Are you saying that $Subject doesn't work, or that the
> > > > above-linked please-review patch doesn't work?
> > > 
> > > shmem pays no attention to the mapping_large_folio_support() flag,
> > > so the proposed fix doesn't work.  It ought to, but it has its own way
> > > of doing it that predates mapping_large_folio_support existing.
> > 
> > Yep.  It turned out to be easier to fix xfile.c to deal with large
> > folios than I thought it would be.  Or so I think.  We'll see what
> > happens on fstestscloud overnight.
> 
> Where do we stand with this?  Should I merge these two patches into
> 6.8-rcX, cc:stable?

This patchset doesn't actually fix the problem, so no, let's not merge
it.

For 6.9 we'll make xfile.c clean w.r.t. large folios:
https://lore.kernel.org/linux-xfs/20240129143502.189370-1-hch@lst.de/

I don't think we need a 6.8 backport since xfile.c is only used by an
experimental feature that is default n in kconfig.

--D