Re: Fwd: btrfs / io-uring corrupting reads

From: Filipe Manana <fdmanana@kernel.org>
To: Daniel Black <daniel@mariadb.org>
Cc: io-uring@vger.kernel.org, linux-btrfs@vger.kernel.org
Subject: Re: Fwd: btrfs / io-uring corrupting reads
Date: Tue, 22 Feb 2022 11:41:56 +0000	[thread overview]
Message-ID: <YhTMBFrZeEvROh0C@debian9.Home> (raw)
In-Reply-To: <CABVffEO3DZTtTNdjkwTegxNPTHbeM-PBeKk5B_dFXdsTvL2wFg@mail.gmail.com>

On Tue, Feb 22, 2022 at 08:53:02AM +1100, Daniel Black wrote:
> Per references at the bottom btfs + iouring manage to corrupt the
> reading of a file.
> 
> Using podman here however docker or another container runtime will
> probably work. As will any MariaDB-10.6 on a distro with a native
> liburing userspace. Apologies for the older and bloated container
> image.
> 
> Reproduction of bug:
> 
> using a btrfs:
> 
> $ dd if=/dev/zero  of=../btrfs.blk bs=1M count=2K
> $ sudo losetup --direct-io=on  -f ../btrfs.blk
> $ sudo mkfs.btrfs /dev/loop6
> $ sudo mount /dev/loop/6 /mnt/btrfstest
> $ sudo mkdir /mnt/btrfstest/noaio
> $ sudo chown dan: /mnt/btrfstest/noaio
> 
> Initialize database on directory:
> 
> $ podman run --name mdbinit --rm -v
> /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
> MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> --innodb_use_native_aio=0
> 
> $ podman kill mdbinit
> 
> Switch to using uring to read:
> 
> $ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
> MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> --innodb_use_native_aio=1
> 
> Failure observed on startup:
> 
> 2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk
> or a failed read of file './ibdata1' page [page id: space=0, page
> number=9]. You may have to recover from a backup.

I gave it a try, but it fails setting up io_uring:

2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool

So that's why it doesn't fail here, as it fallbacks to no aio mode.

Any idea why it's failing to setup io_uring?

I have the liburing2 and liburing-dev packages installed on debian, and
tried with a 5.17-rc4 kernel.

I can run fio with io_uring as the ioengine (works perferctly so far).

Thanks.

> 
> 2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
> 
>  len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB:
> Database page corruption on disk or a failed read of file './ibdata1'
> page [page id: space=0, page number=243]. You may have to recover from
> a backup.
> 
> ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii
> and hex (16384 bytes):
> 
>  len 16384; hex
> 00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21
> 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a
> failed read of file './mysql/innodb_table_stats.ibd' page [page id:
> space=1, page number=0]. You may have to recover from a backup.
> 
> 002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
> 
> Without --innodb_use_native_aio=0 as a container argument this starts
> without error.
> 
> $ sudo losetup --direct-io=off  -f btrfs.blk also exhibits the failure
> 
> Observed failures in:
> * 5.17.0-0.rc4.96.fc36.x86_64
> * 5.16.8 (on nixos)
> * 5.15.6
> 
> No observed failure:
> * 5.15.14-200.fc35.x86_64
> * 5.10
> 
> references:
> * https://jira.mariadb.org/browse/MDEV-27900
> * https://github.com/NixOS/nixpkgs/issues/160516
> * https://jira.mariadb.org/browse/MDEV-27449