Request about "Page cache invalidation failure on direct I/O."

Request about "Page cache invalidation failure on direct I/O."

[Andrea Gelmini]

Hi everybody,
   thanks a lot for your work.

   On my laptop (Ubuntu 19.10, Kernel 5.5.7, VirtualBox 6.1.4-136177)
   I have an SSD (Samsung SSD 860 EVO 4TB + luks + lvm + ext4)
   with a virtual machine, without troubles, since months.

   Now, I move the virtual machine on external USB
   disk (Seagate M3 Portable 4TB + luks + BTRFS).
   Run it, and after a few minutes of simple boot and Windows updating
   (the guest system), I find this in dmesg:

[376827.145222] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
[376827.145225] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
[376827.145230] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
[376827.145231] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
[376827.145234] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
[376827.145234] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
[376827.145236] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
[376827.145237] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
[376827.145240] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
[376827.145241] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0

   I kindly ask your advice. At the moment the virtual seems to work
   without problem.

Thanks a lot,
Gelma

Re: Request about "Page cache invalidation failure on direct I/O."

[David Sterba]

On Wed, Mar 11, 2020 at 08:42:04PM +0000, Andrea Gelmini wrote:
>    On my laptop (Ubuntu 19.10, Kernel 5.5.7, VirtualBox 6.1.4-136177)
>    I have an SSD (Samsung SSD 860 EVO 4TB + luks + lvm + ext4)
>    with a virtual machine, without troubles, since months.
> 
>    Now, I move the virtual machine on external USB
>    disk (Seagate M3 Portable 4TB + luks + BTRFS).
>    Run it, and after a few minutes of simple boot and Windows updating
>    (the guest system), I find this in dmesg:
> 
> [376827.145222] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> [376827.145225] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
> [376827.145230] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> [376827.145231] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
> [376827.145234] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> [376827.145234] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
> [376827.145236] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> [376827.145237] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
> [376827.145240] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> [376827.145241] File: /mnt/4TB/piastrelli/home/virtual/VirtualBox VMs/Zuccotti/Snapshots/{badf36e0-30a3-4fef-b723-4cdab32f2ef0}.vdi PID: 48667 Comm: kworker/1:0
> 
>    I kindly ask your advice. At the moment the virtual seems to work
>    without problem.

The warning is there to point out use of buffered writes and direct io
on the same range. More details are in the commits that added the code,
eg.

5a9d929d6e13278  iomap: report collisions between directio and buffered writes to userspace
a92853b6746fe5f  fs/direct-io.c: keep dio_warn_stale_pagecache() when CONFIG_BLOCK=n

whether the corruption really happens depends. If the collision happens
then fsync will report EIO at some point but if you haven't observed any
problems I think it's ok.

According to the timestamps, the reads happen in a quick sequence.
It could be that something during the VM startup reads parts of the
image as buffered and then it goes only DIO.  Looking to the code, the
message is rate limited so the burst would appear once per day and
without tuning of the rate I can't say if the buffered vs dio happens
all the time or just once. For the 'once' case I would not be worried.

Re: Request about "Page cache invalidation failure on direct I/O."

[Andrea Gelmini]

Il giorno mar 17 mar 2020 alle ore 01:30 David Sterba
<dsterba@suse.cz> ha scritto:
> without tuning of the rate I can't say if the buffered vs dio happens
> all the time or just once. For the 'once' case I would not be worried.

Thanks a lot David for your interest.

Well, with latest kernel (5.5.13), I still see it, but a lot less than
before (same setup, hardware and
software versions).

After a few hours of work I just got this bouch of warnings (btw, no
boot of virtual machine guest, it was in status saved/suspended).

[gio mar 26 05:22:58 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 05:22:58 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 25176 Comm: kworker/4:3
[gio mar 26 05:28:41 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 05:28:41 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 35075 Comm: kworker/4:1
[gio mar 26 05:44:00 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 05:44:00 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 38801 Comm: kworker/4:4
[gio mar 26 05:54:48 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 05:54:48 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 44366 Comm: kworker/4:1
[gio mar 26 06:02:43 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 06:02:43 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 44366 Comm: kworker/4:1
[gio mar 26 06:02:43 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 06:02:43 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 53354 Comm: kworker/4:2
[gio mar 26 06:02:43 2020] Page cache invalidation failure on direct
I/O.  Possible data corruption due to collision with buffered I/O!
[gio mar 26 06:02:43 2020] File:
/mnt/4TB/piastrelli/home/virtual/VirtualBox
VMs/Zuccotti/Snapshots/{23d5aff0-4514-46b9-ab38-76ae59b3acbf}.vdi P
ID: 55712 Comm: kworker/4:6

I have no problem at the moment (this is happening on the test setup
to evaluate BTRFS), but maybe it's important to spot it for you all
devs.
If it help you, I can test the same setup with ext4 or another fs.

Thanks again,
Gelma


Ciao,
Gelma