Re: uring regression - lost write request

[Daniel Black <daniel@mariadb.org>]

> Would it be possible to turn this into a full reproducer script?
> Something that someone that knows nothing about mysqld/mariadb can just
> run and have it reproduce. If I install the 10.6 packages from above,
> then it doesn't seem to use io_uring or be linked against liburing.

Sorry Jens.

Hope containers are ok.

mkdir ~/mdbtest/

$ podman run -d -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -e
MARIADB_USER=sbtest -e MARIADB_PASSWORD=sbtest -e
MARIADB_DATABASE=sbtest  --name mdb10.6-uring_test -v
$HOME/mdbtest:/var/lib/mysql:Z  --security-opt seccomp=unconfined
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_log_file_size=1G  --innodb_buffer_pool_size=50G
--innodb_io_capacity=5000  --innodb_io_capacity_max=9000
--innodb_flush_log_at_trx_commit=0   --innodb_adaptive_flushing_lwm=0
 --innodb-adaptive-flushing=1   --innodb_flush_neighbors=1
--innodb-use-native-aio=1   --innodb_file-per-table=1
--innodb-fast-shutdown=0   --innodb-flush-method=O_DIRECT
--innodb_lru_scan_depth=1024   --innodb_lru_flush_size=256


# drop 50G pool size down if you don't have it. Not critical to
reproduction. IO capacity here should be about what the hardware is.
Otherwise gaps of 0 tps will appear without it being the cause of the
bug.

$ podman logs mdb10.6-uring_test
...
2021-11-11  6:06:49 0 [Warning] innodb_use_native_aio may cause hangs
with this kernel 5.15.0-0.rc7.20211028git1fc596a56b33.56.fc36.x86_64;
see https://jira.mariadb.org/browse/MDEV-26674
2021-11-11  6:06:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-11-11  6:06:49 0 [Note] InnoDB: Number of pools: 1
2021-11-11  6:06:49 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-11-11  6:06:49 0 [Note] mysqld: O_TMPFILE is not supported on
/tmp (disabling future attempts)
2021-11-11  6:06:49 0 [Note] InnoDB: Using liburing

Should contain first and last line here:

$ podman exec  mdb10.6-uring_test sysbench
/usr/share/sysbench/oltp_update_index.lua --mysql-password=sbtest
--percentile=99  --tables=8 --table_size=2000000 prepare

Creating table 'sbtest1'...
Inserting 2000000 records into 'sbtest1'
Creating a secondary index on 'sbtest1'...
Creating table 'sbtest2'...
Inserting 2000000 records into 'sbtest2'
Creating a secondary index on 'sbtest2'...
Creating table 'sbtest3'...
Inserting 2000000 records into 'sbtest3'
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest4'...
Inserting 2000000 records into 'sbtest4'
Creating a secondary index on 'sbtest4'...
Creating table 'sbtest5'...
Inserting 2000000 records into 'sbtest5'
Creating a secondary index on 'sbtest5'...
Creating table 'sbtest6'...
Inserting 2000000 records into 'sbtest6'
Creating a secondary index on 'sbtest6'...
Creating table 'sbtest7'...
Inserting 2000000 records into 'sbtest7'
Creating a secondary index on 'sbtest7'...
Creating table 'sbtest8'...
Inserting 2000000 records into 'sbtest8'
Creating a secondary index on 'sbtest8'...


# Adjust threads there to the amount of hardware threads available.
time is the length of the test.

$ podman exec  mdb10.6-uring_test sysbench
/usr/share/sysbench/oltp_update_index.lua --mysql-password=sbtest
--percentile=99  --tables=8 --table_size=2000000 --rand-seed=42
--rand-type=uniform --max-requests=0 --time=600 --report-interval=5
--threads=64 run



Eventually after
https://mariadb.com/kb/en/innodb-system-variables/#innodb_fatal_semaphore_wait_threshold
of 600 seconds the podman logs mdb10.6-uring_test will contains an
error like:

2021-10-07 17:06:43 0 [ERROR] [FATAL] InnoDB:
innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
211007 17:06:43 [ERROR] mysqld got signal 6 ;


Restarting the container on the same populated ~/mdbtest volume could
be slow due to recovery time. Remove contents and repeat prepare step.

cleanup:

podman kill mdb10.6-uring_test
podman rm mdb10.6-uring_test
sudo rm -rf ~/mdbtest