Re: stalling IO regression since linux 5.12, through 5.18

From: "Chris Murphy" <lists@colorremedies.com>
To: "Jan Kara" <jack@suse.cz>
Cc: "Holger Hoffstätte" <holger@applied-asynchrony.com>,
	"Nikolay Borisov" <nborisov@suse.com>,
	"Jens Axboe" <axboe@kernel.dk>,
	"Paolo Valente" <paolo.valente@linaro.org>,
	Linux-RAID <linux-raid@vger.kernel.org>,
	linux-block <linux-block@vger.kernel.org>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	"Josef Bacik" <josef@toxicpanda.com>
Subject: Re: stalling IO regression since linux 5.12, through 5.18
Date: Wed, 17 Aug 2022 13:57:00 -0400	[thread overview]
Message-ID: <51cd43f9-ab6b-4dd6-814f-e0c1ace3143c@www.fastmail.com> (raw)
In-Reply-To: <f6f899a5-97e2-460f-ad73-73d4b5e38eb6@www.fastmail.com>

On Wed, Aug 17, 2022, at 12:47 PM, Chris Murphy wrote:
Can you boot with
>> "megaraid_sas.host_tagset_enable = 0" kernel option and see whether the
>> issue reproduces?

This has been running an hour without symptoms. It's strongly suggestive, but needs to run overnight to be sure. Anecdotally, the max write IO is less than what I'm used to seeing.

[    0.583121] Kernel command line: BOOT_IMAGE=(md/0)/vmlinuz-5.12.5-300.fc34.x86_64 root=UUID=04f1fb7f-5cc4-4dfb-a7cf-b6b6925bf895 ro rootflags=subvol=root rd.md.uuid=e7782150:092e161a:68395862:31375bca biosdevname=1 net.ifnames=0 log_buf_len=8M plymouth.enable=0 megaraid_sas.host_tagset_enable=0
...
[    6.745964] megasas: 07.714.04.00-rc1
[    6.758472] megaraid_sas 0000:02:00.0: BAR:0x1  BAR's base_addr(phys):0x0000000092000000  mapped virt_addr:0x00000000c54554ff
[    6.758477] megaraid_sas 0000:02:00.0: FW now in Ready state
[    6.770658] megaraid_sas 0000:02:00.0: 63 bit DMA mask and 32 bit consistent mask
[    6.795060] megaraid_sas 0000:02:00.0: firmware supports msix	: (96)
[    6.807537] megaraid_sas 0000:02:00.0: requested/available msix 49/49
[    6.819259] megaraid_sas 0000:02:00.0: current msix/online cpus	: (49/48)
[    6.830800] megaraid_sas 0000:02:00.0: RDPQ mode	: (disabled)
[    6.842031] megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 928	 LDIO threshold: 0
[    6.871246] megaraid_sas 0000:02:00.0: Performance mode :Latency (latency index = 1)
[    6.882265] megaraid_sas 0000:02:00.0: FW supports sync cache	: No
[    6.893034] megaraid_sas 0000:02:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[    6.988550] megaraid_sas 0000:02:00.0: FW provided supportMaxExtLDs: 1	max_lds: 64
[    6.988554] megaraid_sas 0000:02:00.0: controller type	: MR(2048MB)
[    6.988555] megaraid_sas 0000:02:00.0: Online Controller Reset(OCR)	: Enabled
[    6.988556] megaraid_sas 0000:02:00.0: Secure JBOD support	: No
[    6.988557] megaraid_sas 0000:02:00.0: NVMe passthru support	: No
[    6.988558] megaraid_sas 0000:02:00.0: FW provided TM TaskAbort/Reset timeout	: 0 secs/0 secs
[    6.988559] megaraid_sas 0000:02:00.0: JBOD sequence map support	: No
[    6.988560] megaraid_sas 0000:02:00.0: PCI Lane Margining support	: No
[    7.025160] megaraid_sas 0000:02:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[    7.025162] megaraid_sas 0000:02:00.0: INIT adapter done
[    7.025164] megaraid_sas 0000:02:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5707
[    7.029878] megaraid_sas 0000:02:00.0: pci id		: (0x1000)/(0x005d)/(0x1028)/(0x1f47)
[    7.029881] megaraid_sas 0000:02:00.0: unevenspan support	: yes
[    7.029882] megaraid_sas 0000:02:00.0: firmware crash dump	: no
[    7.029883] megaraid_sas 0000:02:00.0: JBOD sequence map	: disabled
[    7.029915] megaraid_sas 0000:02:00.0: Max firmware commands: 927 shared with nr_hw_queues = 1
[    7.029918] scsi host11: Avago SAS based MegaRAID driver

-- 
Chris Murphy