[LSF/MM/BPF TOPIC] Potential silent data loss caused by hwpoisoned page cache

[LSF/MM/BPF TOPIC] Potential silent data loss caused by hwpoisoned page cache

[Yang Shi]

I did fill out the google form but forgot to email to the lists.


When discussing the patch that splits page cache THP in order to offline the
poisoned page, Noaya mentioned there is a bigger problem [1] that prevents this
from working since the page cache page will be truncated if uncorrectable
errors happen.  By looking this deeper it turns out this approach (truncating
poisoned page) may incur silent data loss for all non-readonly filesystems if
the page is dirty. And it may be worse for in-memory filesystem, e.g.
shmem/tmpfs
since the data blocks are actually gone.

To solve this problem we could keep the poisoned dirty page in page cache then
notify the users on any later access, e.g. page fault, read/write, etc.  The
clean page could be truncated as is since they can be reread from disk later on.

The consequence is the filesystems may find poisoned page and manipulate it as
healthy page since all the filesystems actually don't check if the page is
poisoned or not in all the relevant paths except page fault.  In general, we
need to make the filesystems be aware of poisoned page before we could keep the
poisoned page in page cache in order to solve the data loss problem.

To make filesystems be aware of poisoned page we should consider:
- The page should be not written back: clearing dirty flag could prevent from
  writeback.
- The page should not be dropped (it shows as a clean page) by drop caches or
  other callers: the refcount pin from hwpoison could prevent from invalidating
  (called by cache drop, inode cache shrinking, etc), but it doesn't avoid
  invalidation in DIO path.
- The page should be able to get truncated/hole punched/unlinked: it works as it
  is.
- Notify users when the page is accessed, e.g. read/write, page fault and other
  paths (compression, encryption, etc).

The scope of the last one is huge since almost all filesystems need to
do it once
a page is returned from page cache lookup by checking hwpoison flag
for every possible path.

This problem had been slightly discussed before, but seems no action
was taken at that time. [2]

I already converted shmem/tmpfs [3] since it seems more severe for
in-memory filesystem. The hugetlbfs is in-memory filesystem as well,
but it depends on double mapping support for hugeltbfs in order to set
hwpoisoned flag on subpage per the discussion with hugetlb folks.

Regular filesystems are definitely on the list as well. I understand
the problem may be very rare, but it is quite subtle so even if data
corruption is met it is very hard to root to it. So I'd like to
suggest this topic. It is related to both MM and FS, although the
heavy lift is on the FS side.

[1] https://lore.kernel.org/linux-mm/CAHbLzkqNPBh_sK09qfr4yu4WTFOzRy+MKj+PA7iG-adzi9zGsg@mail.gmail.com/T/#m0e959283380156f1d064456af01ae51fdff91265

[2] https://lore.kernel.org/lkml/20210318183350.GT3420@casper.infradead.org/

[3] https://lore.kernel.org/linux-mm/20211020210755.23964-1-shy828301@gmail.com/