Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Thomas Lindroth]

Hi. I recently upgraded to kernel series 5.10 from 4.19 and I now get warnings like
this in dmesg:

page:00000000e966ec4e refcount:1 mapcount:0 mapping:0000000000000000 index:0xd3414 pfn:0x14914a
flags: 0x8000000000000077(locked|referenced|uptodate|lru|active|workingset)
raw: 8000000000000077 ffffdc7f4d312b48 ffffdc7f452452c8 0000000000000000
raw: 00000000000d3414 0000000000000000 00000001ffffffff ffff8fd080123000
page dumped because: fuse: trying to steal weird page

The warning in fuse_check_page() doesn't check for PG_workingset which seems to be what
trips the warning. I'm not entirely sure this is a bogus warning but there used to be
similar bogus warnings caused by a missing PG_waiters check. The PG_workingset
page flag was introduced in 4.20 which explains why I get the warning now.

I only get the new warning if I do writes to a fuse fs (mergerfs) and at the same
time put the system under memory pressure by running many qemu VMs.

/Thomas Lindroth

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Miklos Szeredi]

On Mon, 14 Jun 2021 at 11:56, Thomas Lindroth <thomas.lindroth@gmail.com> wrote:
>
> Hi. I recently upgraded to kernel series 5.10 from 4.19 and I now get warnings like
> this in dmesg:
>
> page:00000000e966ec4e refcount:1 mapcount:0 mapping:0000000000000000 index:0xd3414 pfn:0x14914a
> flags: 0x8000000000000077(locked|referenced|uptodate|lru|active|workingset)
> raw: 8000000000000077 ffffdc7f4d312b48 ffffdc7f452452c8 0000000000000000
> raw: 00000000000d3414 0000000000000000 00000001ffffffff ffff8fd080123000
> page dumped because: fuse: trying to steal weird page
>
> The warning in fuse_check_page() doesn't check for PG_workingset which seems to be what
> trips the warning. I'm not entirely sure this is a bogus warning but there used to be
> similar bogus warnings caused by a missing PG_waiters check. The PG_workingset
> page flag was introduced in 4.20 which explains why I get the warning now.
>
> I only get the new warning if I do writes to a fuse fs (mergerfs) and at the same
> time put the system under memory pressure by running many qemu VMs.

AFAICT fuse is trying to steal a pagecache page from a pipe buffer
created by splice(2).    The page looks okay, but I have no idea what
PG_workingset means in this context.

Matthew, can you please help?

Thanks,
Miklos

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Miklos Szeredi]

On Mon, 14 Jun 2021 at 11:56, Thomas Lindroth <thomas.lindroth@gmail.com> wrote:
>
> Hi. I recently upgraded to kernel series 5.10 from 4.19 and I now get warnings like
> this in dmesg:
>
> page:00000000e966ec4e refcount:1 mapcount:0 mapping:0000000000000000 index:0xd3414 pfn:0x14914a
> flags: 0x8000000000000077(locked|referenced|uptodate|lru|active|workingset)
> raw: 8000000000000077 ffffdc7f4d312b48 ffffdc7f452452c8 0000000000000000
> raw: 00000000000d3414 0000000000000000 00000001ffffffff ffff8fd080123000
> page dumped because: fuse: trying to steal weird page
>
> The warning in fuse_check_page() doesn't check for PG_workingset which seems to be what
> trips the warning. I'm not entirely sure this is a bogus warning but there used to be
> similar bogus warnings caused by a missing PG_waiters check. The PG_workingset
> page flag was introduced in 4.20 which explains why I get the warning now.
>
> I only get the new warning if I do writes to a fuse fs (mergerfs) and at the same
> time put the system under memory pressure by running many qemu VMs.

AFAICT fuse is trying to steal a pagecache page from a pipe buffer
created by splice(2).    The page looks okay, but I have no idea what
PG_workingset means in this context.

Matthew, can you please help?

Thanks,
Miklos

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Matthew Wilcox]

On Wed, Jun 16, 2021 at 02:31:32PM +0200, Miklos Szeredi wrote:
> On Mon, 14 Jun 2021 at 11:56, Thomas Lindroth <thomas.lindroth@gmail.com> wrote:
> >
> > Hi. I recently upgraded to kernel series 5.10 from 4.19 and I now get warnings like
> > this in dmesg:
> >
> > page:00000000e966ec4e refcount:1 mapcount:0 mapping:0000000000000000 index:0xd3414 pfn:0x14914a
> > flags: 0x8000000000000077(locked|referenced|uptodate|lru|active|workingset)
> > raw: 8000000000000077 ffffdc7f4d312b48 ffffdc7f452452c8 0000000000000000
> > raw: 00000000000d3414 0000000000000000 00000001ffffffff ffff8fd080123000
> > page dumped because: fuse: trying to steal weird page
> >
> > The warning in fuse_check_page() doesn't check for PG_workingset which seems to be what
> > trips the warning. I'm not entirely sure this is a bogus warning but there used to be
> > similar bogus warnings caused by a missing PG_waiters check. The PG_workingset
> > page flag was introduced in 4.20 which explains why I get the warning now.
> >
> > I only get the new warning if I do writes to a fuse fs (mergerfs) and at the same
> > time put the system under memory pressure by running many qemu VMs.
> 
> AFAICT fuse is trying to steal a pagecache page from a pipe buffer
> created by splice(2).    The page looks okay, but I have no idea what
> PG_workingset means in this context.
> 
> Matthew, can you please help?

PG_workingset was introduced by Johannes:

    mm: workingset: tell cache transitions from workingset thrashing

    Refaults happen during transitions between workingsets as well as in-place
    thrashing.  Knowing the difference between the two has a range of
    applications, including measuring the impact of memory shortage on the
    system performance, as well as the ability to smarter balance pressure
    between the filesystem cache and the swap-backed workingset.

    During workingset transitions, inactive cache refaults and pushes out
    established active cache.  When that active cache isn't stale, however,
    and also ends up refaulting, that's bonafide thrashing.

    Introduce a new page flag that tells on eviction whether the page has been
    active or not in its lifetime.  This bit is then stored in the shadow
    entry, to classify refaults as transitioning or thrashing.

so I think it's fine for you to ignore when stealing a page.

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Miklos Szeredi]

On Wed, 16 Jun 2021 at 14:59, Matthew Wilcox <willy@infradead.org> wrote:
>
> On Wed, Jun 16, 2021 at 02:31:32PM +0200, Miklos Szeredi wrote:
> > On Mon, 14 Jun 2021 at 11:56, Thomas Lindroth <thomas.lindroth@gmail.com> wrote:
> > >
> > > Hi. I recently upgraded to kernel series 5.10 from 4.19 and I now get warnings like
> > > this in dmesg:
> > >
> > > page:00000000e966ec4e refcount:1 mapcount:0 mapping:0000000000000000 index:0xd3414 pfn:0x14914a
> > > flags: 0x8000000000000077(locked|referenced|uptodate|lru|active|workingset)
> > > raw: 8000000000000077 ffffdc7f4d312b48 ffffdc7f452452c8 0000000000000000
> > > raw: 00000000000d3414 0000000000000000 00000001ffffffff ffff8fd080123000
> > > page dumped because: fuse: trying to steal weird page
> > >
> > > The warning in fuse_check_page() doesn't check for PG_workingset which seems to be what
> > > trips the warning. I'm not entirely sure this is a bogus warning but there used to be
> > > similar bogus warnings caused by a missing PG_waiters check. The PG_workingset
> > > page flag was introduced in 4.20 which explains why I get the warning now.
> > >
> > > I only get the new warning if I do writes to a fuse fs (mergerfs) and at the same
> > > time put the system under memory pressure by running many qemu VMs.
> >
> > AFAICT fuse is trying to steal a pagecache page from a pipe buffer
> > created by splice(2).    The page looks okay, but I have no idea what
> > PG_workingset means in this context.
> >
> > Matthew, can you please help?
>
> PG_workingset was introduced by Johannes:
>
>     mm: workingset: tell cache transitions from workingset thrashing
>
>     Refaults happen during transitions between workingsets as well as in-place
>     thrashing.  Knowing the difference between the two has a range of
>     applications, including measuring the impact of memory shortage on the
>     system performance, as well as the ability to smarter balance pressure
>     between the filesystem cache and the swap-backed workingset.
>
>     During workingset transitions, inactive cache refaults and pushes out
>     established active cache.  When that active cache isn't stale, however,
>     and also ends up refaulting, that's bonafide thrashing.
>
>     Introduce a new page flag that tells on eviction whether the page has been
>     active or not in its lifetime.  This bit is then stored in the shadow
>     entry, to classify refaults as transitioning or thrashing.
>
> so I think it's fine for you to ignore when stealing a page.

I have problem understanding what a workingset is.  Is it related to
swap?  If so, how can such a page be part of a file mapping?

Thanks,
Miklos

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Miklos Szeredi]

On Wed, 16 Jun 2021 at 14:59, Matthew Wilcox <willy@infradead.org> wrote:
>
> On Wed, Jun 16, 2021 at 02:31:32PM +0200, Miklos Szeredi wrote:
> > On Mon, 14 Jun 2021 at 11:56, Thomas Lindroth <thomas.lindroth@gmail.com> wrote:
> > >
> > > Hi. I recently upgraded to kernel series 5.10 from 4.19 and I now get warnings like
> > > this in dmesg:
> > >
> > > page:00000000e966ec4e refcount:1 mapcount:0 mapping:0000000000000000 index:0xd3414 pfn:0x14914a
> > > flags: 0x8000000000000077(locked|referenced|uptodate|lru|active|workingset)
> > > raw: 8000000000000077 ffffdc7f4d312b48 ffffdc7f452452c8 0000000000000000
> > > raw: 00000000000d3414 0000000000000000 00000001ffffffff ffff8fd080123000
> > > page dumped because: fuse: trying to steal weird page
> > >
> > > The warning in fuse_check_page() doesn't check for PG_workingset which seems to be what
> > > trips the warning. I'm not entirely sure this is a bogus warning but there used to be
> > > similar bogus warnings caused by a missing PG_waiters check. The PG_workingset
> > > page flag was introduced in 4.20 which explains why I get the warning now.
> > >
> > > I only get the new warning if I do writes to a fuse fs (mergerfs) and at the same
> > > time put the system under memory pressure by running many qemu VMs.
> >
> > AFAICT fuse is trying to steal a pagecache page from a pipe buffer
> > created by splice(2).    The page looks okay, but I have no idea what
> > PG_workingset means in this context.
> >
> > Matthew, can you please help?
>
> PG_workingset was introduced by Johannes:
>
>     mm: workingset: tell cache transitions from workingset thrashing
>
>     Refaults happen during transitions between workingsets as well as in-place
>     thrashing.  Knowing the difference between the two has a range of
>     applications, including measuring the impact of memory shortage on the
>     system performance, as well as the ability to smarter balance pressure
>     between the filesystem cache and the swap-backed workingset.
>
>     During workingset transitions, inactive cache refaults and pushes out
>     established active cache.  When that active cache isn't stale, however,
>     and also ends up refaulting, that's bonafide thrashing.
>
>     Introduce a new page flag that tells on eviction whether the page has been
>     active or not in its lifetime.  This bit is then stored in the shadow
>     entry, to classify refaults as transitioning or thrashing.
>
> so I think it's fine for you to ignore when stealing a page.

I have problem understanding what a workingset is.  Is it related to
swap?  If so, how can such a page be part of a file mapping?

Thanks,
Miklos

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Vlastimil Babka]

On 6/16/21 3:47 PM, Miklos Szeredi wrote:
> On Wed, 16 Jun 2021 at 14:59, Matthew Wilcox <willy@infradead.org> wrote:
>>
>> > AFAICT fuse is trying to steal a pagecache page from a pipe buffer
>> > created by splice(2).    The page looks okay, but I have no idea what
>> > PG_workingset means in this context.
>> >
>> > Matthew, can you please help?
>>
>> PG_workingset was introduced by Johannes:
>>
>>     mm: workingset: tell cache transitions from workingset thrashing
>>
>>     Refaults happen during transitions between workingsets as well as in-place
>>     thrashing.  Knowing the difference between the two has a range of
>>     applications, including measuring the impact of memory shortage on the
>>     system performance, as well as the ability to smarter balance pressure
>>     between the filesystem cache and the swap-backed workingset.
>>
>>     During workingset transitions, inactive cache refaults and pushes out
>>     established active cache.  When that active cache isn't stale, however,
>>     and also ends up refaulting, that's bonafide thrashing.
>>
>>     Introduce a new page flag that tells on eviction whether the page has been
>>     active or not in its lifetime.  This bit is then stored in the shadow
>>     entry, to classify refaults as transitioning or thrashing.
>>
>> so I think it's fine for you to ignore when stealing a page.
> 
> I have problem understanding what a workingset is.  Is it related to

"working set" is the notion of the set of pages that the workload needs to
access at the moment/relatively short time window, and it would be beneficial if
all of it could fit in the RAM.
PG_workinsgset is part of the mechanism that tries to estimate this ideal set of
pages, and especially when the workload shifts to another set of pages, in order
to guide reclaim better. See the big comment at the top of mm/workingset.c for
details

> swap?  If so, how can such a page be part of a file mapping?

Not related to swap. It was actually first implemented only for file pages (page
cache), but then extended to anonymous pages by aae466b0052e ("mm/swap:
implement workingset detection for anonymous LRU")

> Thanks,
> Miklos
> 

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Johannes Weiner]

On Wed, Jun 16, 2021 at 06:48:34PM +0200, Vlastimil Babka wrote:
> On 6/16/21 3:47 PM, Miklos Szeredi wrote:
> > On Wed, 16 Jun 2021 at 14:59, Matthew Wilcox <willy@infradead.org> wrote:
> >>
> >> > AFAICT fuse is trying to steal a pagecache page from a pipe buffer
> >> > created by splice(2).    The page looks okay, but I have no idea what
> >> > PG_workingset means in this context.
> >> >
> >> > Matthew, can you please help?
> >>
> >> PG_workingset was introduced by Johannes:
> >>
> >>     mm: workingset: tell cache transitions from workingset thrashing
> >>
> >>     Refaults happen during transitions between workingsets as well as in-place
> >>     thrashing.  Knowing the difference between the two has a range of
> >>     applications, including measuring the impact of memory shortage on the
> >>     system performance, as well as the ability to smarter balance pressure
> >>     between the filesystem cache and the swap-backed workingset.
> >>
> >>     During workingset transitions, inactive cache refaults and pushes out
> >>     established active cache.  When that active cache isn't stale, however,
> >>     and also ends up refaulting, that's bonafide thrashing.
> >>
> >>     Introduce a new page flag that tells on eviction whether the page has been
> >>     active or not in its lifetime.  This bit is then stored in the shadow
> >>     entry, to classify refaults as transitioning or thrashing.
> >>
> >> so I think it's fine for you to ignore when stealing a page.
> > 
> > I have problem understanding what a workingset is.  Is it related to
> 
> "working set" is the notion of the set of pages that the workload needs to
> access at the moment/relatively short time window, and it would be beneficial if
> all of it could fit in the RAM.
> PG_workinsgset is part of the mechanism that tries to estimate this ideal set of
> pages, and especially when the workload shifts to another set of pages, in order
> to guide reclaim better. See the big comment at the top of mm/workingset.c for
> details
> 
> > swap?  If so, how can such a page be part of a file mapping?
> 
> Not related to swap. It was actually first implemented only for file pages (page
> cache), but then extended to anonymous pages by aae466b0052e ("mm/swap:
> implement workingset detection for anonymous LRU")

Thanks, yes.

Think of it as similar to PG_active. It's just another usage/heat
indicator of file and anon pages on the reclaim LRU that, unlike
PG_active, persists across deactivation and even reclaim (we store it
in the page cache / swapper cache tree until the page refaults).

So if fuse accepts pages that can legally have PG_active set,
PG_workingset is fine too.

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Miklos Szeredi]

On Fri, 18 Jun 2021 at 19:29, Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Wed, Jun 16, 2021 at 06:48:34PM +0200, Vlastimil Babka wrote:
> > On 6/16/21 3:47 PM, Miklos Szeredi wrote:
> > > On Wed, 16 Jun 2021 at 14:59, Matthew Wilcox <willy@infradead.org> wrote:
> > >>
> > >> > AFAICT fuse is trying to steal a pagecache page from a pipe buffer
> > >> > created by splice(2).    The page looks okay, but I have no idea what
> > >> > PG_workingset means in this context.
> > >> >
> > >> > Matthew, can you please help?
> > >>
> > >> PG_workingset was introduced by Johannes:
> > >>
> > >>     mm: workingset: tell cache transitions from workingset thrashing
> > >>
> > >>     Refaults happen during transitions between workingsets as well as in-place
> > >>     thrashing.  Knowing the difference between the two has a range of
> > >>     applications, including measuring the impact of memory shortage on the
> > >>     system performance, as well as the ability to smarter balance pressure
> > >>     between the filesystem cache and the swap-backed workingset.
> > >>
> > >>     During workingset transitions, inactive cache refaults and pushes out
> > >>     established active cache.  When that active cache isn't stale, however,
> > >>     and also ends up refaulting, that's bonafide thrashing.
> > >>
> > >>     Introduce a new page flag that tells on eviction whether the page has been
> > >>     active or not in its lifetime.  This bit is then stored in the shadow
> > >>     entry, to classify refaults as transitioning or thrashing.
> > >>
> > >> so I think it's fine for you to ignore when stealing a page.
> > >
> > > I have problem understanding what a workingset is.  Is it related to
> >
> > "working set" is the notion of the set of pages that the workload needs to
> > access at the moment/relatively short time window, and it would be beneficial if
> > all of it could fit in the RAM.
> > PG_workinsgset is part of the mechanism that tries to estimate this ideal set of
> > pages, and especially when the workload shifts to another set of pages, in order
> > to guide reclaim better. See the big comment at the top of mm/workingset.c for
> > details
> >
> > > swap?  If so, how can such a page be part of a file mapping?
> >
> > Not related to swap. It was actually first implemented only for file pages (page
> > cache), but then extended to anonymous pages by aae466b0052e ("mm/swap:
> > implement workingset detection for anonymous LRU")
>
> Thanks, yes.
>
> Think of it as similar to PG_active. It's just another usage/heat
> indicator of file and anon pages on the reclaim LRU that, unlike
> PG_active, persists across deactivation and even reclaim (we store it
> in the page cache / swapper cache tree until the page refaults).
>
> So if fuse accepts pages that can legally have PG_active set,
> PG_workingset is fine too.

Thanks, fix pushed to:

  git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#for-next

Miklos

Re: Possible bogus "fuse: trying to steal weird page" warning related to PG_workingset.

[Miklos Szeredi]

On Fri, 18 Jun 2021 at 19:29, Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Wed, Jun 16, 2021 at 06:48:34PM +0200, Vlastimil Babka wrote:
> > On 6/16/21 3:47 PM, Miklos Szeredi wrote:
> > > On Wed, 16 Jun 2021 at 14:59, Matthew Wilcox <willy@infradead.org> wrote:
> > >>
> > >> > AFAICT fuse is trying to steal a pagecache page from a pipe buffer
> > >> > created by splice(2).    The page looks okay, but I have no idea what
> > >> > PG_workingset means in this context.
> > >> >
> > >> > Matthew, can you please help?
> > >>
> > >> PG_workingset was introduced by Johannes:
> > >>
> > >>     mm: workingset: tell cache transitions from workingset thrashing
> > >>
> > >>     Refaults happen during transitions between workingsets as well as in-place
> > >>     thrashing.  Knowing the difference between the two has a range of
> > >>     applications, including measuring the impact of memory shortage on the
> > >>     system performance, as well as the ability to smarter balance pressure
> > >>     between the filesystem cache and the swap-backed workingset.
> > >>
> > >>     During workingset transitions, inactive cache refaults and pushes out
> > >>     established active cache.  When that active cache isn't stale, however,
> > >>     and also ends up refaulting, that's bonafide thrashing.
> > >>
> > >>     Introduce a new page flag that tells on eviction whether the page has been
> > >>     active or not in its lifetime.  This bit is then stored in the shadow
> > >>     entry, to classify refaults as transitioning or thrashing.
> > >>
> > >> so I think it's fine for you to ignore when stealing a page.
> > >
> > > I have problem understanding what a workingset is.  Is it related to
> >
> > "working set" is the notion of the set of pages that the workload needs to
> > access at the moment/relatively short time window, and it would be beneficial if
> > all of it could fit in the RAM.
> > PG_workinsgset is part of the mechanism that tries to estimate this ideal set of
> > pages, and especially when the workload shifts to another set of pages, in order
> > to guide reclaim better. See the big comment at the top of mm/workingset.c for
> > details
> >
> > > swap?  If so, how can such a page be part of a file mapping?
> >
> > Not related to swap. It was actually first implemented only for file pages (page
> > cache), but then extended to anonymous pages by aae466b0052e ("mm/swap:
> > implement workingset detection for anonymous LRU")
>
> Thanks, yes.
>
> Think of it as similar to PG_active. It's just another usage/heat
> indicator of file and anon pages on the reclaim LRU that, unlike
> PG_active, persists across deactivation and even reclaim (we store it
> in the page cache / swapper cache tree until the page refaults).
>
> So if fuse accepts pages that can legally have PG_active set,
> PG_workingset is fine too.

Thanks, fix pushed to:

  git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#for-next

Miklos