Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Julian Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 3220 bytes --]

Dear Maintainer,

when running Linux Kernel version 6.0.12, 6.0.10, 6.0-rc7, or 6.1.4, my 
system seemingly randomly freezes due to the file system being set to 
read-only due to an issue with my NVMe controller.
The issue does *not* appear on Linux Kernel version 5.19.11 or lower.

Through network logging I am able to catch the issue:
```
Jan  8 14:50:16 x299-desktop kernel: [ 1461.259288] nvme nvme0: 
controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Does 
your device have a faulty power saving mode enabled?
Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Try 
"nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
Jan  8 14:50:16 x299-desktop kernel: [ 1461.331360] nvme 0000:01:00.0: 
enabling device (0000 -> 0002)
Jan  8 14:50:16 x299-desktop kernel: [ 1461.331458] nvme nvme0: Removing 
after probe failure status: -19
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371389] nvme0n1: detected 
capacity change from 1953525168 to 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371389] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371389] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371392] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371394] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371405] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371406] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371411] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371419] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371425] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Jan  8 14:50:16 x299-desktop kernel: [ 1461.371426] BTRFS error (device 
nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 11, rd 0, flush 0, corrupt 0, gen 0
```

I have tried the suggestion in the log without luck.

Attached is a log that includes two system freezes, as well as a list of 
PCI(e) devices created by Debian reportbug.
The first freeze happens at "Jan  8 04:26:28" and the second freeze 
happens at "Jan  8 14:50:16".

Currently, I am using git bisect to narrow down the window of possible 
commits, but since the issue appears seemingly random, it will take many 
months to identify the offending commit this way.

The original Debian bug report is here: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028309


Greetings
Julian Groß

[-- Attachment #1.1.2: x299-desktop_crash.log.xz --]
[-- Type: application/x-xz, Size: 245400 bytes --]

[-- Attachment #1.1.3: pci_devices.txt --]
[-- Type: text/plain, Size: 63200 bytes --]

** PCI devices:
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Sky Lake-E DMI3 Registers [1458:5000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 0
	NUMA node: 0
	IOMMU group: 0
	Capabilities: <access denied>

00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Sky Lake-E CBDMA Registers [1458:8888]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 98
	NUMA node: 0
	IOMMU group: 1
	Region 0: Memory at 3800fff40000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 100
	NUMA node: 0
	IOMMU group: 2
	Region 0: Memory at 3800fff3c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 98
	NUMA node: 0
	IOMMU group: 3
	Region 0: Memory at 3800fff38000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 100
	NUMA node: 0
	IOMMU group: 4
	Region 0: Memory at 3800fff34000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 98
	NUMA node: 0
	IOMMU group: 5
	Region 0: Memory at 3800fff30000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 100
	NUMA node: 0
	IOMMU group: 6
	Region 0: Memory at 3800fff2c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 98
	NUMA node: 0
	IOMMU group: 7
	Region 0: Memory at 3800fff28000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 100
	NUMA node: 0
	IOMMU group: 8
	Region 0: Memory at 3800fff24000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 9
	Capabilities: <access denied>

00:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS [8086:2025] (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 10
	Capabilities: <access denied>

00:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOAPIC [8086:2026] (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Sky Lake-E IOAPIC [8086:2026]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	NUMA node: 0
	IOMMU group: 11
	Region 0: Memory at 92f28000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>

00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Sky Lake-E Ubox Registers [1458:5000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 12
	Capabilities: <access denied>

00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Ubox Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 13

00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Ubox Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 14
	Capabilities: <access denied>

00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af] (prog-if 30 [XHCI])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [1458:5007]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 36
	NUMA node: 0
	IOMMU group: 15
	Region 0: Memory at 3800fff10000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH CSME HECI [1458:1c3a]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 104
	NUMA node: 0
	IOMMU group: 16
	Region 0: Memory at 3800fff45000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: mei_me
	Kernel modules: mei_me

00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282] (prog-if 01 [AHCI 1.0])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH SATA controller [AHCI mode] [1458:b005]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 38
	NUMA node: 0
	IOMMU group: 17
	Region 0: Memory at 92f24000 (32-bit, non-prefetchable) [size=8K]
	Region 1: Memory at 92f27000 (32-bit, non-prefetchable) [size=256]
	Region 2: I/O ports at 3050 [size=8]
	Region 3: I/O ports at 3040 [size=4]
	Region 4: I/O ports at 3020 [size=32]
	Region 5: Memory at 92f26000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: <access denied>
	Kernel driver in use: ahci
	Kernel modules: ahci

00:1b.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #17 [8086:a2e7] (rev f0) (prog-if 00 [Normal decode])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH PCI Express Root Port [1458:5001]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 24
	NUMA node: 0
	IOMMU group: 18
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: 92e00000-92efffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1b.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #21 [8086:a2eb] (rev f0) (prog-if 00 [Normal decode])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH PCI Express Root Port [1458:5001]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	IOMMU group: 19
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: 92d00000-92dfffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0) (prog-if 00 [Normal decode])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH PCI Express Root Port [1458:5001]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 26
	NUMA node: 0
	IOMMU group: 20
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1c.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #5 [8086:a294] (rev f0) (prog-if 00 [Normal decode])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH PCI Express Root Port [1458:5001]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 27
	NUMA node: 0
	IOMMU group: 21
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: 92c00000-92cfffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1d.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #9 [8086:a298] (rev f0) (prog-if 00 [Normal decode])
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH PCI Express Root Port [1458:5001]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 28
	NUMA node: 0
	IOMMU group: 22
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: 92b00000-92bfffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1f.0 ISA bridge [0601]: Intel Corporation X299 Chipset LPC/eSPI Controller [8086:a2d2]
	Subsystem: Gigabyte Technology Co., Ltd X299 Chipset LPC/eSPI Controller [1458:5001]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0
	IOMMU group: 23

00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
	Subsystem: Gigabyte Technology Co., Ltd 200 Series/Z370 Chipset Family Power Management Controller [1458:5001]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 23
	Region 0: Memory at 92f20000 (32-bit, non-prefetchable) [disabled] [size=16K]

00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
	Subsystem: Gigabyte Technology Co., Ltd 200 Series PCH HD Audio [1458:a182]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 110
	NUMA node: 0
	IOMMU group: 23
	Region 0: Memory at 3800fff20000 (64-bit, non-prefetchable) [size=16K]
	Region 4: Memory at 3800fff00000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
	Subsystem: Gigabyte Technology Co., Ltd 200 Series/Z370 Chipset Family SMBus Controller [1458:5001]
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	IOMMU group: 23
	Region 0: Memory at 3800fff44000 (64-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at 3000 [size=32]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801

00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
	Subsystem: Gigabyte Technology Co., Ltd Ethernet Connection (2) I219-V [1458:e000]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 37
	NUMA node: 0
	IOMMU group: 24
	Region 0: Memory at 92f00000 (32-bit, non-prefetchable) [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: e1000e
	Kernel modules: e1000e

01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808] (prog-if 02 [NVM Express])
	Subsystem: Samsung Electronics Co Ltd SSD 970 EVO [144d:a801]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	IOMMU group: 25
	Region 0: Memory at 92e00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: nvme
	Kernel modules: nvme

02:00.0 Audio device [0403]: Creative Labs CA0132 Sound Core3D [Sound Blaster Recon3D / Z-Series / Sound BlasterX AE-5 Plus] [1102:0012] (rev 01)
	Subsystem: Creative Labs SB1570 SB Audigy Fx [1102:0010]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	IOMMU group: 26
	Region 0: Memory at 92d04000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at 92d00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142/ASM3142 USB 3.1 Host Controller [1b21:2142] (prog-if 30 [XHCI])
	Subsystem: Gigabyte Technology Co., Ltd ASM2142/ASM3142 USB 3.1 Host Controller [1458:5007]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	IOMMU group: 27
	Region 0: Memory at 92c00000 (64-bit, non-prefetchable) [size=32K]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808] (prog-if 02 [NVM Express])
	Subsystem: Samsung Electronics Co Ltd SSD 970 EVO [144d:a801]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	IOMMU group: 28
	Region 0: Memory at 92b00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: nvme
	Kernel modules: nvme

16:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04) (prog-if 00 [Normal decode])
	Subsystem: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:0000]
	Physical Slot: 4
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 30
	NUMA node: 0
	IOMMU group: 29
	Bus: primary=16, secondary=17, subordinate=17, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: b5e00000-b5efffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

16:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E VT-d [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 30
	Capabilities: <access denied>

16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 31
	Capabilities: <access denied>

16:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	NUMA node: 0
	IOMMU group: 32
	Region 0: Memory at b5f00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>

16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 33

16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 34

16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 34

16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 35

16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 36

16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 36

16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 37

16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 37

16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 37

16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E CHA Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 37

16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E PCU Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 38

17:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808] (prog-if 02 [NVM Express])
	Subsystem: Samsung Electronics Co Ltd SSD 970 EVO [144d:a801]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 49
	NUMA node: 0
	IOMMU group: 39
	Region 0: Memory at b5e00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: nvme
	Kernel modules: nvme

64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04) (prog-if 00 [Normal decode])
	Subsystem: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:0000]
	Physical Slot: 6
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 32
	NUMA node: 0
	IOMMU group: 40
	Bus: primary=64, secondary=65, subordinate=65, sec-latency=0
	I/O behind bridge: b000-bfff [size=4K] [16-bit]
	Memory behind bridge: d7000000-d80fffff [size=17M] [32-bit]
	Prefetchable memory behind bridge: 3802e0000000-3802f1ffffff [size=288M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

64:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E VT-d [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 41
	Capabilities: <access denied>

64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 42
	Capabilities: <access denied>

64:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	NUMA node: 0
	IOMMU group: 43
	Region 0: Memory at d8100000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>

64:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2066] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 44
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2066] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 45
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0a.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2040] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 46
	Capabilities: <access denied>

64:0a.1 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2041] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 47
	Capabilities: <access denied>

64:0a.2 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2042] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 48
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0a.3 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2043] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 49
	Capabilities: <access denied>

64:0a.4 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2044] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 50
	Capabilities: <access denied>

64:0a.5 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 1 [8086:2045] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LM Channel 1 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 51
	Capabilities: <access denied>

64:0a.6 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 1 [8086:2046] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMS Channel 1 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 52
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0a.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 53
	Capabilities: <access denied>

64:0b.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E DECS Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 54
	Capabilities: <access denied>

64:0b.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LM Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 55
	Capabilities: <access denied>

64:0b.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMS Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 56
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0b.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 57
	Capabilities: <access denied>

64:0c.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2040] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 58
	Capabilities: <access denied>

64:0c.1 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2041] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 59
	Capabilities: <access denied>

64:0c.2 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2042] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 60
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0c.3 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2043] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 61
	Capabilities: <access denied>

64:0c.4 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2044] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 62
	Capabilities: <access denied>

64:0c.5 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 1 [8086:2045] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LM Channel 1 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 63
	Capabilities: <access denied>

64:0c.6 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 1 [8086:2046] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMS Channel 1 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 64
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0c.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 65
	Capabilities: <access denied>

64:0d.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E DECS Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 66
	Capabilities: <access denied>

64:0d.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LM Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 67
	Capabilities: <access denied>

64:0d.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMS Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 68
	Capabilities: <access denied>
	Kernel driver in use: skx_uncore

64:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 69
	Capabilities: <access denied>

65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Gigabyte Technology Co., Ltd GP104 [GeForce GTX 1080] [1458:3702]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 112
	NUMA node: 0
	IOMMU group: 70
	Region 0: Memory at d7000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at 3802e0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at 3802f0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at b000 [size=128]
	Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: nvidia
	Kernel modules: nvidia

65:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd GP104 High Definition Audio Controller [1458:3702]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin B routed to IRQ 111
	NUMA node: 0
	IOMMU group: 70
	Region 0: Memory at d8080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04) (prog-if 00 [Normal decode])
	Subsystem: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:0000]
	Physical Slot: 0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 34
	NUMA node: 0
	IOMMU group: 71
	Bus: primary=b2, secondary=b3, subordinate=b3, sec-latency=0
	I/O behind bridge: [disabled] [16-bit]
	Memory behind bridge: fbb00000-fbefffff [size=4M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

b2:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E VT-d [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 72
	Capabilities: <access denied>

b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 73
	Capabilities: <access denied>

b2:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	NUMA node: 0
	IOMMU group: 74
	Region 0: Memory at fbf00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>

b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 75
	Capabilities: <access denied>

b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 76
	Kernel driver in use: skx_uncore

b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 76
	Kernel driver in use: skx_uncore

b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E M2PCI Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 77

b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E M2PCI Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 78

b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Intel Corporation Sky Lake-E M2PCI Registers [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 78

b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	IOMMU group: 79

b3:00.0 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 35
	NUMA node: 0
	IOMMU group: 80
	Region 0: Memory at fbebc000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbe60000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbe40000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbc40000 [disabled] [size=256K]
	Capabilities: <access denied>
	Kernel driver in use: be2net
	Kernel modules: be2net

b3:00.1 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin B routed to IRQ 94
	NUMA node: 0
	IOMMU group: 81
	Region 0: Memory at fbeb8000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbe20000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbe00000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbc00000 [disabled] [size=256K]
	Capabilities: <access denied>
	Kernel driver in use: be2net
	Kernel modules: be2net

b3:00.2 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	!!! Unknown header type 7f
	Interrupt: pin ? routed to IRQ 98
	NUMA node: 0
	IOMMU group: 82
	Region 0: Memory at fbeb4000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbde0000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbdc0000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbe90000 [disabled] [size=64K]
	Kernel modules: be2net

b3:00.3 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	!!! Unknown header type 7f
	Interrupt: pin ? routed to IRQ 98
	NUMA node: 0
	IOMMU group: 83
	Region 0: Memory at fbeb0000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbda0000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbd80000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbe80000 [disabled] [size=64K]
	Kernel modules: be2net

b3:00.4 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	!!! Unknown header type 7f
	Interrupt: pin ? routed to IRQ 35
	NUMA node: 0
	IOMMU group: 84
	Region 0: Memory at fbeac000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbd60000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbd40000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbbc0000 [disabled] [size=256K]
	Kernel modules: be2net

b3:00.5 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	!!! Unknown header type 7f
	Interrupt: pin ? routed to IRQ 94
	NUMA node: 0
	IOMMU group: 85
	Region 0: Memory at fbea8000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbd20000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbd00000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbb80000 [disabled] [size=256K]
	Kernel modules: be2net

b3:00.6 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	!!! Unknown header type 7f
	Interrupt: pin ? routed to IRQ 98
	NUMA node: 0
	IOMMU group: 86
	Region 0: Memory at fbea4000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbce0000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbcc0000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbb40000 [disabled] [size=256K]
	Kernel modules: be2net

b3:00.7 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
	Subsystem: Hewlett-Packard Company NC552SFP 2-port 10Gb Server Adapter [103c:3340]
	!!! Unknown header type 7f
	Interrupt: pin ? routed to IRQ 98
	NUMA node: 0
	IOMMU group: 87
	Region 0: Memory at fbea0000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fbca0000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at fbc80000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at fbb00000 [disabled] [size=256K]
	Kernel modules: be2net

[-- Attachment #1.1.4: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3197 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Linux kernel regression tracking (Thorsten Leemhuis)]

[adding the nvme maintainers and the regressions mailing list to the
list of recipients]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 11.01.23 23:11, Julian Groß wrote:
> Dear Maintainer,
> 
> when running Linux Kernel version 6.0.12, 6.0.10, 6.0-rc7, or 6.1.4, my
> system seemingly randomly freezes due to the file system being set to
> read-only due to an issue with my NVMe controller.
> The issue does *not* appear on Linux Kernel version 5.19.11 or lower.
> 
> Through network logging I am able to catch the issue:
> ```
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259288] nvme nvme0:
> controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Does
> your device have a faulty power saving mode enabled?
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Try
> "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.331360] nvme 0000:01:00.0:
> enabling device (0000 -> 0002)
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.331458] nvme nvme0: Removing
> after probe failure status: -19
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371389] nvme0n1: detected
> capacity change from 1953525168 to 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371389] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371389] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371392] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371394] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371405] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371406] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371411] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371419] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371425] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 10, rd 0, flush 0, corrupt 0,
> gen 0
> Jan  8 14:50:16 x299-desktop kernel: [ 1461.371426] BTRFS error (device
> nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 11, rd 0, flush 0, corrupt 0,
> gen 0
> ```
> 
> I have tried the suggestion in the log without luck.
> 
> Attached is a log that includes two system freezes, as well as a list of
> PCI(e) devices created by Debian reportbug.
> The first freeze happens at "Jan  8 04:26:28" and the second freeze
> happens at "Jan  8 14:50:16".
> 
> Currently, I am using git bisect to narrow down the window of possible
> commits, but since the issue appears seemingly random, it will take many
> months to identify the offending commit this way.
> 
> The original Debian bug report is here:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028309

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced v5.19..v6.0-rc7
#regzbot title nvme: system partially freezes with "nvme controller is down"
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Keith Busch]

On Wed, Jan 11, 2023 at 10:11:22PM +0000, Julian Groß wrote:
> 
> Currently, I am using git bisect to narrow down the window of possible
> commits, but since the issue appears seemingly random, it will take many
> months to identify the offending commit this way.

Unfortunately bisect may be our best option. Link loss like that usually
is a problem below the nvme driver level.

Re: Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Bjorn Helgaas]

On Thu, Jan 12, 2023 at 03:48:46PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
> ...
> On 11.01.23 23:11, Julian Groß wrote:
> > Dear Maintainer,
> > 
> > when running Linux Kernel version 6.0.12, 6.0.10, 6.0-rc7, or 6.1.4, my
> > system seemingly randomly freezes due to the file system being set to
> > read-only due to an issue with my NVMe controller.
> > The issue does *not* appear on Linux Kernel version 5.19.11 or lower.
> > 
> > Through network logging I am able to catch the issue:
> > ```
> > Jan  8 14:50:16 x299-desktop kernel: [ 1461.259288] nvme nvme0:
> > controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
> > Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Does
> > your device have a faulty power saving mode enabled?
> > Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Try
> > "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
> > Jan  8 14:50:16 x299-desktop kernel: [ 1461.331360] nvme 0000:01:00.0:
> > enabling device (0000 -> 0002)
> > ...
> > 
> > I have tried the suggestion in the log without luck.
> > 
> > Attached is a log that includes two system freezes, as well as a list of
> > PCI(e) devices created by Debian reportbug.
> > The first freeze happens at "Jan  8 04:26:28" and the second freeze
> > happens at "Jan  8 14:50:16".
> > 
> > Currently, I am using git bisect to narrow down the window of possible
> > commits, but since the issue appears seemingly random, it will take many
> > months to identify the offending commit this way.
> > 
> > The original Debian bug report is here:
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028309

For some reason the log [1] has very little of the kernel dmesg log.
It does seem like the freeze is partial (I see messages for hundreds
or thousands of seconds after the nvme reset), but requires a reboot
to recover.

The lspci information [2] shows the 00:1b.0 Root Port leading to the
01:00.0 NVMe device.

Is it possible to collect lspci output after the nvme freeze?  If so,
please save the output of:

  sudo lspci -vv -s00:1b.0
  sudo lspci -vv -s01:00.0

Make sure to run lspci as root so we can see the error logging
registers for these devices.

If you can collect more of the dmesg log after the freeze, e.g., via
the "dmesg" command, that might be helpful, too.

Bjorn

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1028309;filename=x299-desktop_crash.log.xz;msg=5
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=0;bug=1028309;msg=5

Re: Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Linux regression tracking (Thorsten Leemhuis)]

Hi, this is your Linux kernel regression tracker. Top-posting for once,
to make this easily accessible to everyone.

I might be missing something, but it looks like this discussion stalled.
I wonder why.

Julian, did you ever share the data Bjorn asked for? Or tried a a
bisection, as suggested by Keith? Or did you stop caring for some
reason? Does everything maybe work fine these days?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

On 12.01.23 17:42, Bjorn Helgaas wrote:
> On Thu, Jan 12, 2023 at 03:48:46PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
>> ...
>> On 11.01.23 23:11, Julian Groß wrote:
>>> Dear Maintainer,
>>>
>>> when running Linux Kernel version 6.0.12, 6.0.10, 6.0-rc7, or 6.1.4, my
>>> system seemingly randomly freezes due to the file system being set to
>>> read-only due to an issue with my NVMe controller.
>>> The issue does *not* appear on Linux Kernel version 5.19.11 or lower.
>>>
>>> Through network logging I am able to catch the issue:
>>> ```
>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259288] nvme nvme0:
>>> controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Does
>>> your device have a faulty power saving mode enabled?
>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Try
>>> "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.331360] nvme 0000:01:00.0:
>>> enabling device (0000 -> 0002)
>>> ...
>>>
>>> I have tried the suggestion in the log without luck.
>>>
>>> Attached is a log that includes two system freezes, as well as a list of
>>> PCI(e) devices created by Debian reportbug.
>>> The first freeze happens at "Jan  8 04:26:28" and the second freeze
>>> happens at "Jan  8 14:50:16".
>>>
>>> Currently, I am using git bisect to narrow down the window of possible
>>> commits, but since the issue appears seemingly random, it will take many
>>> months to identify the offending commit this way.
>>>
>>> The original Debian bug report is here:
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028309
> 
> For some reason the log [1] has very little of the kernel dmesg log.
> It does seem like the freeze is partial (I see messages for hundreds
> or thousands of seconds after the nvme reset), but requires a reboot
> to recover.
> 
> The lspci information [2] shows the 00:1b.0 Root Port leading to the
> 01:00.0 NVMe device.
> 
> Is it possible to collect lspci output after the nvme freeze?  If so,
> please save the output of:
> 
>   sudo lspci -vv -s00:1b.0
>   sudo lspci -vv -s01:00.0
> 
> Make sure to run lspci as root so we can see the error logging
> registers for these devices.
> 
> If you can collect more of the dmesg log after the freeze, e.g., via
> the "dmesg" command, that might be helpful, too.
> 
> Bjorn
> 
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1028309;filename=x299-desktop_crash.log.xz;msg=5
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=0;bug=1028309;msg=5
> 
> 

Re: Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Julian]

Hello,

yes it is partially stalled. The reason being that I have run into the issue on a 5.x kernel.
The issue sometimes takes days to reproduce, but I now think that it might be a hardware issue.
Whatever the case, it isn't a regression in 6.x.

I couldn't really do anything that Bjorn suggested, since the system is unresponsive when the freeze issue happens. I am amazed that network logging still works.

Greetings
Julian Groß

17.02.2023 13:40:02 Linux regression tracking (Thorsten Leemhuis) <regressions@leemhuis.info>:

> Hi, this is your Linux kernel regression tracker. Top-posting for once,
> to make this easily accessible to everyone.
> 
> I might be missing something, but it looks like this discussion stalled.
> I wonder why.
> 
> Julian, did you ever share the data Bjorn asked for? Or tried a a
> bisection, as suggested by Keith? Or did you stop caring for some
> reason? Does everything maybe work fine these days?
> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
> 
> #regzbot poke
> 
> On 12.01.23 17:42, Bjorn Helgaas wrote:
>> On Thu, Jan 12, 2023 at 03:48:46PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
>>> ...
>>> On 11.01.23 23:11, Julian Groß wrote:
>>>> Dear Maintainer,
>>>> 
>>>> when running Linux Kernel version 6.0.12, 6.0.10, 6.0-rc7, or 6.1.4, my
>>>> system seemingly randomly freezes due to the file system being set to
>>>> read-only due to an issue with my NVMe controller.
>>>> The issue does *not* appear on Linux Kernel version 5.19.11 or lower.
>>>> 
>>>> Through network logging I am able to catch the issue:
>>>> ```
>>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259288] nvme nvme0:
>>>> controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
>>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Does
>>>> your device have a faulty power saving mode enabled?
>>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.259293] nvme nvme0: Try
>>>> "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
>>>> Jan  8 14:50:16 x299-desktop kernel: [ 1461.331360] nvme 0000:01:00.0:
>>>> enabling device (0000 -> 0002)
>>>> ...
>>>> 
>>>> I have tried the suggestion in the log without luck.
>>>> 
>>>> Attached is a log that includes two system freezes, as well as a list of
>>>> PCI(e) devices created by Debian reportbug.
>>>> The first freeze happens at "Jan  8 04:26:28" and the second freeze
>>>> happens at "Jan  8 14:50:16".
>>>> 
>>>> Currently, I am using git bisect to narrow down the window of possible
>>>> commits, but since the issue appears seemingly random, it will take many
>>>> months to identify the offending commit this way.
>>>> 
>>>> The original Debian bug report is here:
>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028309
>> 
>> For some reason the log [1] has very little of the kernel dmesg log.
>> It does seem like the freeze is partial (I see messages for hundreds
>> or thousands of seconds after the nvme reset), but requires a reboot
>> to recover.
>> 
>> The lspci information [2] shows the 00:1b.0 Root Port leading to the
>> 01:00.0 NVMe device.
>> 
>> Is it possible to collect lspci output after the nvme freeze?  If so,
>> please save the output of:
>> 
>>   sudo lspci -vv -s00:1b.0
>>   sudo lspci -vv -s01:00.0
>> 
>> Make sure to run lspci as root so we can see the error logging
>> registers for these devices.
>> 
>> If you can collect more of the dmesg log after the freeze, e.g., via
>> the "dmesg" command, that might be helpful, too.
>> 
>> Bjorn
>> 
>> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1028309;filename=x299-desktop_crash.log.xz;msg=5
>> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=0;bug=1028309;msg=5
>> 
>> 

Re: Regression in Kernel 6.0: System partially freezes with "nvme controller is down"

[Linux regression tracking #update (Thorsten Leemhuis)]

On 12.01.23 15:48, Linux kernel regression tracking (Thorsten Leemhuis)
wrote:
> On 11.01.23 23:11, Julian Groß wrote:
>>
>> when running Linux Kernel version 6.0.12, 6.0.10, 6.0-rc7, or 6.1.4, my
>> system seemingly randomly freezes due to the file system being set to
>> read-only due to an issue with my NVMe controller.
>> The issue does *not* appear on Linux Kernel version 5.19.11 or lower.
>>
>> Through network logging I am able to catch the issue:
> 
> [...]
> 
> #regzbot ^introduced v5.19..v6.0-rc7
> #regzbot title nvme: system partially freezes with "nvme controller is down"
> #regzbot ignore-activity

Stop tracking this for now:

#regzbot inconclusive: stalled and might be a hw issue
#regzbot ignore-activity

For details see:

https://lore.kernel.org/all/81b5b28e-33fb-48ca-9e84-7574d5596bfb@posteo.de/

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.