Re: [PATCHSET v5 0/12] Add support for async buffered reads

From: Andres Freund <andres@anarazel.de>
To: Jens Axboe <axboe@kernel.dk>
Cc: io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	akpm@linux-foundation.org
Subject: Re: [PATCHSET v5 0/12] Add support for async buffered reads
Date: Fri, 5 Jun 2020 12:56:17 -0700	[thread overview]
Message-ID: <20200605195617.26wwsvoo4cvnmdum@alap3.anarazel.de> (raw)
In-Reply-To: <20200604013045.7gu7xopreusbdea2@alap3.anarazel.de>

Hi,

On 2020-06-03 18:30:45 -0700, Andres Freund wrote:
> > I'll try and reproduce this, any chance you have a test case that can
> > be run so I don't have to write one from scratch? The more detailed
> > instructions the better.
>
> It shouldn't be too hard to write you a detailed script for reproducing
> the issue. But it'd not be an all that minimal reproducer, unless it
> also triggers on smaller scale (it's a 130GB database that triggers the
> problem reliably, and small tables don't seem to do so reliably).
>
> I'll try to write that up after I set up kvm / repro there.

Sorry, took a bit longer. Ran into some issues with my aio branch when
running on 5.6. Not yet sure if that's my fault, or io_uring's.

Here's a recipe that repros reliably for me. For me it triggers reliably
at this size (-s 1000 ends up around ~16GB).

# Build aio branch of postgres
sudo mkdir /srv/src
sudo chown andres. /srv/src
cd /srv/src
git clone -b aio https://github.com/anarazel/postgres.git pg
cd pg
./configure --enable-depend --with-liburing --prefix=/home/andres/bin/
make -j48 -s install

# check that target directory is in PATH
type psql
psql is /home/andres/bin/bin/psql

in one shell:
# create database directory, start postgres
echo 1200 | sudo tee /proc/sys/vm/nr_hugepages
initdb /srv/pguringcrash
postgres -D /srv/pguringcrash/ -c bgwriter_delay=10ms -c bgwriter_lru_maxpages=1000 -c wal_level=minimal -c max_wal_senders=0 -c wal_buffers=128MB -c max_wal_size=100GB -c shared_buffers=2GB -c max_parallel_workers_per_gather=8 -c huge_pages=on

in the other shell:
# fill with test data (skipping index creation etc)
pgbench -i -s 1000 postgres -IdtGv -q

# and trigger a parallel sequential scan, using 8 additional workers
echo 3 | sudo tee /proc/sys/vm/drop_caches && psql -c 'SET max_parallel_workers_per_gather=8' -c 'SELECT SUM(abalance) FROM pgbench_accounts ;'

For me that faily reliably triggers the issue within a few seconds.

[  323.312854] BUG: unable to handle page fault for address: 00007f0f1428c000
[  323.319879] #PF: supervisor read access in kernel mode
[  323.321391] #PF: error_code(0x0001) - permissions violation
[  323.323019] PGD 102e760067 P4D 102e760067 PUD 1037878067 PMD 8000000fc7e000e7
[  323.325065] Oops: 0001 [#1] SMP NOPTI
[  323.326248] CPU: 1 PID: 3145 Comm: postgres Not tainted 5.7.0-rc7-andres-00133-gc8707bf69395 #43
[  323.328653] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1 04/01/2014
[  323.330365] RIP: 0010:__wake_up_common+0x53/0x140
[  323.331276] Code: 41 f6 01 04 0f 85 a3 00 00 00 48 8b 43 08 48 8d 78 e8 48 8d 43 08 48 89 c1 48 89 04 24 48 8d 47 18 48 39 c1 0f 84 b8 00 00 00 <48> 8b 47 18 89 54 24 14 31 ed 4c 8d 60 e8 8b 1f f6 c3 04 75 52 48
[  323.334419] RSP: 0018:ffffc90000dcbdc8 EFLAGS: 00010086
[  323.335409] RAX: 00007f0f1428c000 RBX: ffff889038b189f8 RCX: ffff889038b18a00
[  323.336723] RDX: 0000000000000001 RSI: 0000000000000003 RDI: 00007f0f1428bfe8
[  323.338035] RBP: 0000000000000246 R08: 0000000000000001 R09: ffffc90000dcbe18
[  323.339331] R10: ffff889036a5e750 R11: 0000000000000000 R12: 0000000000000001
[  323.340312] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000001
[  323.341204] FS:  00007f0f6f271740(0000) GS:ffff88903f840000(0000) knlGS:0000000000000000
[  323.342217] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  323.342950] CR2: 00007f0f1428c000 CR3: 0000001037ed0005 CR4: 0000000000760ee0
[  323.343836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  323.344724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  323.345613] PKRU: 55555554
[  323.346017] Call Trace:
[  323.346435]  __wake_up_common_lock+0x7c/0xc0
[  323.347015]  ep_free+0x30/0xd0
[  323.347468]  ep_eventpoll_release+0x16/0x20
[  323.348040]  __fput+0xda/0x240
[  323.348498]  task_work_run+0x62/0x90
[  323.349015]  exit_to_usermode_loop+0xbd/0xe0
[  323.349573]  do_syscall_64+0xf2/0x130
[  323.349987]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  323.350523] RIP: 0033:0x7f0f6f5a5f13
[  323.350906] Code: 00 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8
[  323.352501] RSP: 002b:00007ffdf3eab5a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[  323.353277] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f0f6f5a5f13
[  323.353947] RDX: 0000000000000001 RSI: 000055ef9f5cb360 RDI: 000000000000002a
[  323.354587] RBP: 000055ef9f5cb2e8 R08: 00000000000280e8 R09: 0000000000000143
[  323.355227] R10: 000000000000000a R11: 0000000000000246 R12: 000055ef9f5cb2e8
[  323.355872] R13: 0000000000000008 R14: 00000000ffffffff R15: 00007f0f6d3407a4
[  323.356510] Modules linked in: 9pnet_virtio isst_if_common iTCO_wdt 9pnet xhci_pci iTCO_vendor_support xhci_hcd
[  323.357477] CR2: 00007f0f1428c000
[  323.357866] ---[ end trace 081898b3c6b5ab1b ]---

I'll try your new branch next.

Greetings,

Andres Freund