* btrfs / io-uring corrupting reads @ 2022-02-21 5:15 Daniel Black 2022-02-21 21:53 ` Fwd: " Daniel Black 0 siblings, 1 reply; 8+ messages in thread From: Daniel Black @ 2022-02-21 5:15 UTC (permalink / raw) To: io-uring, linux-btrfs Per references at the bottom btfs + iouring manage to corrupt the reading of a file. Using podman here however docker or another container runtime will probably work. As will any MariaDB-10.6 on a distro with a native liburing userspace. Apologies for the older and bloated container image. Reproduction of bug: using a btrfs: $ dd if=/dev/zero of=../btrfs.blk bs=1M count=2K $ sudo losetup --direct-io=on -f ../btrfs.blk $ sudo mkfs.btrfs /dev/loop6 $ sudo mount /dev/loop/6 /mnt/btrfstest $ sudo mkdir /mnt/btrfstest/noaio $ sudo chown dan: /mnt/btrfstest/noaio Initialize database on directory: $ podman run --name mdbinit --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/danielgblack/mariadb-test:10.6-impish-sysbench --innodb_use_native_aio=0 $ podman kill mdbinit Switch to using uring to read: $ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/danielgblack/mariadb-test:10.6-impish-sysbench --innodb_use_native_aio=1 Failure observed on startup: 2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=9]. You may have to recover from a backup. 2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=243]. You may have to recover from a backup. ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/innodb_table_stats.ibd' page [page id: space=1, page number=0]. You may have to recover from a backup. 002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): Without --innodb_use_native_aio=0 as a container argument this starts without error. $ sudo losetup --direct-io=off -f btrfs.blk also exhibits the failure Observed failures in: * 5.17.0-0.rc4.96.fc36.x86_64 * 5.16.8 (on nixos) * 5.15.6 No observed failure: * 5.15.14-200.fc35.x86_64 * 5.10 references: * https://jira.mariadb.org/browse/MDEV-27900 * https://github.com/NixOS/nixpkgs/issues/160516 * https://jira.mariadb.org/browse/MDEV-27449 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Fwd: btrfs / io-uring corrupting reads 2022-02-21 5:15 btrfs / io-uring corrupting reads Daniel Black @ 2022-02-21 21:53 ` Daniel Black 2022-02-22 11:41 ` Filipe Manana 0 siblings, 1 reply; 8+ messages in thread From: Daniel Black @ 2022-02-21 21:53 UTC (permalink / raw) To: io-uring, linux-btrfs Per references at the bottom btfs + iouring manage to corrupt the reading of a file. Using podman here however docker or another container runtime will probably work. As will any MariaDB-10.6 on a distro with a native liburing userspace. Apologies for the older and bloated container image. Reproduction of bug: using a btrfs: $ dd if=/dev/zero of=../btrfs.blk bs=1M count=2K $ sudo losetup --direct-io=on -f ../btrfs.blk $ sudo mkfs.btrfs /dev/loop6 $ sudo mount /dev/loop/6 /mnt/btrfstest $ sudo mkdir /mnt/btrfstest/noaio $ sudo chown dan: /mnt/btrfstest/noaio Initialize database on directory: $ podman run --name mdbinit --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/danielgblack/mariadb-test:10.6-impish-sysbench --innodb_use_native_aio=0 $ podman kill mdbinit Switch to using uring to read: $ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/danielgblack/mariadb-test:10.6-impish-sysbench --innodb_use_native_aio=1 Failure observed on startup: 2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=9]. You may have to recover from a backup. 2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=243]. You may have to recover from a backup. ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/innodb_table_stats.ibd' page [page id: space=1, page number=0]. You may have to recover from a backup. 002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): Without --innodb_use_native_aio=0 as a container argument this starts without error. $ sudo losetup --direct-io=off -f btrfs.blk also exhibits the failure Observed failures in: * 5.17.0-0.rc4.96.fc36.x86_64 * 5.16.8 (on nixos) * 5.15.6 No observed failure: * 5.15.14-200.fc35.x86_64 * 5.10 references: * https://jira.mariadb.org/browse/MDEV-27900 * https://github.com/NixOS/nixpkgs/issues/160516 * https://jira.mariadb.org/browse/MDEV-27449 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: btrfs / io-uring corrupting reads 2022-02-21 21:53 ` Fwd: " Daniel Black @ 2022-02-22 11:41 ` Filipe Manana 2022-02-22 12:22 ` Daniel Black 0 siblings, 1 reply; 8+ messages in thread From: Filipe Manana @ 2022-02-22 11:41 UTC (permalink / raw) To: Daniel Black; +Cc: io-uring, linux-btrfs On Tue, Feb 22, 2022 at 08:53:02AM +1100, Daniel Black wrote: > Per references at the bottom btfs + iouring manage to corrupt the > reading of a file. > > Using podman here however docker or another container runtime will > probably work. As will any MariaDB-10.6 on a distro with a native > liburing userspace. Apologies for the older and bloated container > image. > > Reproduction of bug: > > using a btrfs: > > $ dd if=/dev/zero of=../btrfs.blk bs=1M count=2K > $ sudo losetup --direct-io=on -f ../btrfs.blk > $ sudo mkfs.btrfs /dev/loop6 > $ sudo mount /dev/loop/6 /mnt/btrfstest > $ sudo mkdir /mnt/btrfstest/noaio > $ sudo chown dan: /mnt/btrfstest/noaio > > Initialize database on directory: > > $ podman run --name mdbinit --rm -v > /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > --innodb_use_native_aio=0 > > $ podman kill mdbinit > > Switch to using uring to read: > > $ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > --innodb_use_native_aio=1 > > Failure observed on startup: > > 2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk > or a failed read of file './ibdata1' page [page id: space=0, page > number=9]. You may have to recover from a backup. I gave it a try, but it fails setting up io_uring: 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool So that's why it doesn't fail here, as it fallbacks to no aio mode. Any idea why it's failing to setup io_uring? I have the liburing2 and liburing-dev packages installed on debian, and tried with a 5.17-rc4 kernel. I can run fio with io_uring as the ioengine (works perferctly so far). Thanks. > > 2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): > > len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB: > Database page corruption on disk or a failed read of file './ibdata1' > page [page id: space=0, page number=243]. You may have to recover from > a backup. > > ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii > and hex (16384 bytes): > > len 16384; hex > 00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21 > 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a > failed read of file './mysql/innodb_table_stats.ibd' page [page id: > space=1, page number=0]. You may have to recover from a backup. > > 002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): > > Without --innodb_use_native_aio=0 as a container argument this starts > without error. > > $ sudo losetup --direct-io=off -f btrfs.blk also exhibits the failure > > Observed failures in: > * 5.17.0-0.rc4.96.fc36.x86_64 > * 5.16.8 (on nixos) > * 5.15.6 > > No observed failure: > * 5.15.14-200.fc35.x86_64 > * 5.10 > > references: > * https://jira.mariadb.org/browse/MDEV-27900 > * https://github.com/NixOS/nixpkgs/issues/160516 > * https://jira.mariadb.org/browse/MDEV-27449 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: btrfs / io-uring corrupting reads 2022-02-22 11:41 ` Filipe Manana @ 2022-02-22 12:22 ` Daniel Black 2022-02-22 12:46 ` Filipe Manana 0 siblings, 1 reply; 8+ messages in thread From: Daniel Black @ 2022-02-22 12:22 UTC (permalink / raw) To: Filipe Manana; +Cc: io-uring, linux-btrfs On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote: > I gave it a try, but it fails setting up io_uring: > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool > > So that's why it doesn't fail here, as it fallbacks to no aio mode. error 1 is EPERM. Seems it needs --privileged on the container startup as a podman argument (before the image name). Sorry I missed that > Any idea why it's failing to setup io_uring? > > I have the liburing2 and liburing-dev packages installed on debian, and > tried with a 5.17-rc4 kernel. Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package: mariadb-install-db --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=0 mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1 should achieve the same thing. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: btrfs / io-uring corrupting reads 2022-02-22 12:22 ` Daniel Black @ 2022-02-22 12:46 ` Filipe Manana 2022-02-22 12:54 ` Filipe Manana 0 siblings, 1 reply; 8+ messages in thread From: Filipe Manana @ 2022-02-22 12:46 UTC (permalink / raw) To: Daniel Black; +Cc: io-uring, linux-btrfs On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote: > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > I gave it a try, but it fails setting up io_uring: > > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool > > > > So that's why it doesn't fail here, as it fallbacks to no aio mode. > > error 1 is EPERM. Seems it needs --privileged on the container startup > as a podman argument (before the image name). Sorry I missed that > > > Any idea why it's failing to setup io_uring? > > > > I have the liburing2 and liburing-dev packages installed on debian, and > > tried with a 5.17-rc4 kernel. > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package: > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path > --innodb-use-native-aio=0 > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1 > > should achieve the same thing. Sorry, I have no experience with mariadb and podman. How am I supposed to run that? Is that supposed to run inside the container, on the host? Do I need to change the podman command lines? What I did before was: DEV=/dev/sdh MNT=/mnt/sdh mkfs.btrfs -f $DEV mount $DEV $MNT mkdir $MNT/noaio chown fdmanana: $MNT/noaio podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/danielgblack/mariadb-test:10.6-impish-sysbench --innodb_use_native_aio=0 Then in another shell: podman kill --all podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/danielgblack/mariadb-test:10.6-impish-sysbench --innodb_use_native_aio=1 What should I change or add in there? Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: btrfs / io-uring corrupting reads 2022-02-22 12:46 ` Filipe Manana @ 2022-02-22 12:54 ` Filipe Manana 2022-03-01 21:51 ` Daniel Black 0 siblings, 1 reply; 8+ messages in thread From: Filipe Manana @ 2022-02-22 12:54 UTC (permalink / raw) To: Daniel Black; +Cc: io-uring, linux-btrfs On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote: > > On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote: > > > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > > > I gave it a try, but it fails setting up io_uring: > > > > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) > > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 > > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool > > > > > > So that's why it doesn't fail here, as it fallbacks to no aio mode. > > > > error 1 is EPERM. Seems it needs --privileged on the container startup > > as a podman argument (before the image name). Sorry I missed that > > > > > Any idea why it's failing to setup io_uring? > > > > > > I have the liburing2 and liburing-dev packages installed on debian, and > > > tried with a 5.17-rc4 kernel. > > > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package: > > > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path > > --innodb-use-native-aio=0 > > > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1 > > > > should achieve the same thing. > > Sorry, I have no experience with mariadb and podman. How am I supposed > to run that? > Is that supposed to run inside the container, on the host? Do I need > to change the podman command lines? > > What I did before was: > > DEV=/dev/sdh > MNT=/mnt/sdh > > mkfs.btrfs -f $DEV > mount $DEV $MNT > > mkdir $MNT/noaio > chown fdmanana: $MNT/noaio > > podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > --innodb_use_native_aio=0 > > > Then in another shell: > > podman kill --all > > podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > --innodb_use_native_aio=1 > > > What should I change or add in there? Ok, just passing --privileged to both podman commands triggered the bug as in your report. I'll see if I can figure out what's causing the read corruption. > > Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: btrfs / io-uring corrupting reads 2022-02-22 12:54 ` Filipe Manana @ 2022-03-01 21:51 ` Daniel Black 2022-03-02 11:28 ` Filipe Manana 0 siblings, 1 reply; 8+ messages in thread From: Daniel Black @ 2022-03-01 21:51 UTC (permalink / raw) To: Filipe Manana; +Cc: io-uring, linux-btrfs Filipe, DId you find anything? This is starting to be noticed by our mutual users. https://jira.mariadb.org/browse/MDEV-27900 https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Corrupt.20database.20page.20when.20updating.20from.2010.2E5 On Tue, Feb 22, 2022 at 11:55 PM Filipe Manana <fdmanana@kernel.org> wrote: > > On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > > On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote: > > > > > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > > > > > I gave it a try, but it fails setting up io_uring: > > > > > > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) > > > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 > > > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool > > > > > > > > So that's why it doesn't fail here, as it fallbacks to no aio mode. > > > > > > error 1 is EPERM. Seems it needs --privileged on the container startup > > > as a podman argument (before the image name). Sorry I missed that > > > > > > > Any idea why it's failing to setup io_uring? > > > > > > > > I have the liburing2 and liburing-dev packages installed on debian, and > > > > tried with a 5.17-rc4 kernel. > > > > > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package: > > > > > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path > > > --innodb-use-native-aio=0 > > > > > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1 > > > > > > should achieve the same thing. > > > > Sorry, I have no experience with mariadb and podman. How am I supposed > > to run that? > > Is that supposed to run inside the container, on the host? Do I need > > to change the podman command lines? > > > > What I did before was: > > > > DEV=/dev/sdh > > MNT=/mnt/sdh > > > > mkfs.btrfs -f $DEV > > mount $DEV $MNT > > > > mkdir $MNT/noaio > > chown fdmanana: $MNT/noaio > > > > podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > > --innodb_use_native_aio=0 > > > > > > Then in another shell: > > > > podman kill --all > > > > podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > > --innodb_use_native_aio=1 > > > > > > What should I change or add in there? > > Ok, just passing --privileged to both podman commands triggered the > bug as in your report. > I'll see if I can figure out what's causing the read corruption. > > > > > > Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: btrfs / io-uring corrupting reads 2022-03-01 21:51 ` Daniel Black @ 2022-03-02 11:28 ` Filipe Manana 0 siblings, 0 replies; 8+ messages in thread From: Filipe Manana @ 2022-03-02 11:28 UTC (permalink / raw) To: Daniel Black; +Cc: io-uring, linux-btrfs On Wed, Mar 02, 2022 at 08:51:53AM +1100, Daniel Black wrote: > Filipe, > > DId you find anything? This is starting to be noticed by our mutual users. Yes, I've sent a patch to address it: https://lore.kernel.org/linux-btrfs/39c96b5608ed99b7d666d4d159f8d135e86b9606.1646219178.git.fdmanana@suse.com/T/#u The details are in the changelog. Basically, MariaDB is getting a short read - it asks to read 16K of data but it only gets the first 4K of data (or the first 8K or 12K). It treats such partial read as a corruption, but in fact it could check that it's a short read and then try to read the remaining data. I understand it's an unexpected result, since it knows its read requests don't cross the EOF boundary and knows exactly the length and file offsets of its pages/blocks, so it expects a read requesting 16K of data to return exactly 16K of data, and no less than that. I've worked in the database industry before, and the same assumptions existed on the engines I worked on. Either way it was a behaviour change in btrfs and it can, and should be, addressed in btrfs. And yes, before your report, there was at least another on the btrfs mailing list from someone getting MariaDB corruptions on btrfs only. However yours had a useful reproducer that made it easier to dig into. Thanks. > > https://jira.mariadb.org/browse/MDEV-27900 > https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Corrupt.20database.20page.20when.20updating.20from.2010.2E5 > > On Tue, Feb 22, 2022 at 11:55 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > > On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > > > > On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote: > > > > > > > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote: > > > > > > > > > I gave it a try, but it fails setting up io_uring: > > > > > > > > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) > > > > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 > > > > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF > > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 > > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool > > > > > > > > > > So that's why it doesn't fail here, as it fallbacks to no aio mode. > > > > > > > > error 1 is EPERM. Seems it needs --privileged on the container startup > > > > as a podman argument (before the image name). Sorry I missed that > > > > > > > > > Any idea why it's failing to setup io_uring? > > > > > > > > > > I have the liburing2 and liburing-dev packages installed on debian, and > > > > > tried with a 5.17-rc4 kernel. > > > > > > > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package: > > > > > > > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path > > > > --innodb-use-native-aio=0 > > > > > > > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1 > > > > > > > > should achieve the same thing. > > > > > > Sorry, I have no experience with mariadb and podman. How am I supposed > > > to run that? > > > Is that supposed to run inside the container, on the host? Do I need > > > to change the podman command lines? > > > > > > What I did before was: > > > > > > DEV=/dev/sdh > > > MNT=/mnt/sdh > > > > > > mkfs.btrfs -f $DEV > > > mount $DEV $MNT > > > > > > mkdir $MNT/noaio > > > chown fdmanana: $MNT/noaio > > > > > > podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e > > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > > > --innodb_use_native_aio=0 > > > > > > > > > Then in another shell: > > > > > > podman kill --all > > > > > > podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e > > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > > > --innodb_use_native_aio=1 > > > > > > > > > What should I change or add in there? > > > > Ok, just passing --privileged to both podman commands triggered the > > bug as in your report. > > I'll see if I can figure out what's causing the read corruption. > > > > > > > > > > Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-03-02 11:28 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-02-21 5:15 btrfs / io-uring corrupting reads Daniel Black 2022-02-21 21:53 ` Fwd: " Daniel Black 2022-02-22 11:41 ` Filipe Manana 2022-02-22 12:22 ` Daniel Black 2022-02-22 12:46 ` Filipe Manana 2022-02-22 12:54 ` Filipe Manana 2022-03-01 21:51 ` Daniel Black 2022-03-02 11:28 ` Filipe Manana
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).