Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 6576 bytes --]

Hi all,

I get kernel panic on 'floppy' module load. If I blacklist the module,
then everything works.
The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
fine. PV dom0 works too. I haven't tried bare metal, but I assume it
works fine too.
The issue happens on newer kernels too (5.8.10 crashes, 5.8.5 works),
but I have it most analyzed on 4.19.x.

Other interesting observations:
- The crash does not manifest when the VM does nothing else than loading
  floppy (by adding systemd.target=modprobe@floppy.service to kernel
  cmdline).
- This bug is not just another "memory corruption" one, but appears to
  be caused by a hole where the IOAPIC registers are supposed to be
  mapped. If this is a pagetable corruption then this is a blocker.
- No other significant differences in kmsg are observed (In particular,
  I/O APIC appears to be initialised successfully without any
  anomalies).
- Modules linked in: floppy(+) xenfs xen_gntdev xen_gntalloc xen_privcmd
  xen_evtchn overlay xen_blkfront
- Loading floppy early in boot along with the module list above (nopat
  modules-load=xenfs,xen-gntdev,xen-gntalloc,xen-privcmd,xen-evtchn,xen-blkfront,overlay,floppy)
  seems to avoid the crash.
- Even with the above, unloading and reloading floppy still results in a crash.

I've read git log between 4.19.142 and 4.19.143 and I don't see any
obvious change that could cause this. The only thing that may be remotely
relevant is a "XEN uses irqdesc::irq_data_common::handler_data to store
a per interrupt XEN data pointer which contains XEN specific
information." commit, so I'm adding involved people to this thread.

The actual crash message (this is from 4.19.144, but it is the same on
4.19.143):

[    2.631097] BUG: unable to handle kernel paging request at ffffffffff5f9000
[    2.631117] PGD 1220e067 P4D 1220e067 PUD 12210067 PMD 12211067 PTE 0
[    2.631135] Oops: 0002 [#1] SMP PTI
[    2.631147] CPU: 1 PID: 275 Comm: systemd-udevd Tainted: G           O      4.19.144-1.pvops.qubes.x86_64 #1
[    2.631173] Hardware name: Xen HVM domU, BIOS 4.8.5-22.fc25 08/15/2020
[    2.631192] RIP: 0010:ioapic_configure_entry+0x66/0xb0
[    2.631206] Code: 8d 88 04 02 00 00 48 8d 04 c0 44 8d 52 11 4d 8d 1c c0 c1 e1 0c 48 63 c9 41 8b 43 14 25 ff 0f 00 00 48 2d 00 10 80 00 48 29 c8 <44> 89 10 44 89 48 10 41 8b 43 14 83 c2 10
 25 ff 0f 00 00 48 2d 00
[    2.631251] RSP: 0000:ffffaf4f40353b08 EFLAGS: 00010086
[    2.631265] RAX: ffffffffff5f9000 RBX: ffff8c3155ad7480 RCX: 0000000000206000
[    2.631285] RDX: 0000000000000008 RSI: ffff8c314e369640 RDI: 00000000ffffffff
[    2.631304] RBP: ffff8c3155914828 R08: ffffffffb09cfac0 R09: 0000000000000001
[    2.631324] R10: 0000000000000019 R11: ffffffffb09cfb50 R12: ffff8c3155935e00
[    2.631343] R13: ffff8c3155914828 R14: ffff8c3155914960 R15: ffff8c31559148a4
[    2.631363] FS:  000072ea2b67fb80(0000) GS:ffff8c3156f00000(0000) knlGS:0000000000000000
[    2.631383] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.631399] CR2: ffffffffff5f9000 CR3: 000000000c266005 CR4: 00000000001606e0
[    2.631420] Call Trace:
[    2.631433]  mp_irqdomain_activate+0x21/0x40
[    2.631448]  __irq_domain_activate_irq+0x60/0xa0
[    2.631462]  irq_domain_activate_irq+0x25/0x40
[    2.631476]  __setup_irq+0x3ba/0x720
[    2.631486]  request_threaded_irq+0xfa/0x170
[    2.631502]  floppy_module_init+0xa33/0x1d23 [floppy]
[    2.631518]  ? netlink_broadcast_filtered+0x157/0x410
[    2.631533]  ? set_cmos+0x112/0x112 [floppy]
[    2.631548]  do_one_initcall+0x4d/0x1d6
[    2.631559]  ? free_unref_page_commit+0x9f/0x120
[    2.631573]  ? _cond_resched+0x16/0x40
[    2.631584]  ? kmem_cache_alloc_trace+0x169/0x1e0
[    2.631598]  do_init_module+0x5b/0x20e
[    2.631610]  load_module+0x1bb9/0x1fc0
[    2.631624]  ? ima_post_read_file+0xe2/0x120
[    2.631639]  ? __do_sys_finit_module+0xd2/0x100
[    2.631653]  __do_sys_finit_module+0xd2/0x100
[    2.631667]  do_syscall_64+0x5b/0x190
[    2.631679]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[    2.631693] RIP: 0033:0x72ea2c7a137d
[    2.631704] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d eb 6a
 0c 00 f7 d8 64 89 01 48
[    2.631749] RSP: 002b:00007fffafe7b4f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[    2.631770] RAX: ffffffffffffffda RBX: 000064f1749ff120 RCX: 000072ea2c7a137d
[    2.631790] RDX: 0000000000000000 RSI: 000072ea2c40095d RDI: 0000000000000006
[    2.631809] RBP: 0000000000020000 R08: 0000000000000000 R09: 0000000000000007
[    2.631828] R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000000000
[    2.631848] R13: 000072ea2c40095d R14: 000064f174c89690 R15: 000064f1749fedd0
[    2.631868] Modules linked in: floppy(+) iwldvm(+) mac80211 fjes(-) iwlwifi cfg80211 ttm ehci_pci rfkill e1000e(+) drm_kms_helper ehci_hcd ata_generic pata_acpi i2c_piix4 u2mfn(O) xen_gnt
dev xen_gntalloc xen_blkback xen_evtchn drm xenfs xen_privcmd overlay xen_blkfront
[    2.631932] CR2: ffffffffff5f9000
[    2.631944] ---[ end trace 28ae492a4a502a7c ]---
[    2.631958] RIP: 0010:ioapic_configure_entry+0x66/0xb0
[    2.631972] Code: 8d 88 04 02 00 00 48 8d 04 c0 44 8d 52 11 4d 8d 1c c0 c1 e1 0c 48 63 c9 41 8b 43 14 25 ff 0f 00 00 48 2d 00 10 80 00 48 29 c8 <44> 89 10 44 89 48 10 41 8b 43 14 83 c2 10
 25 ff 0f 00 00 48 2d 00
[    2.632018] RSP: 0000:ffffaf4f40353b08 EFLAGS: 00010086
[    2.632032] RAX: ffffffffff5f9000 RBX: ffff8c3155ad7480 RCX: 0000000000206000
[    2.632051] RDX: 0000000000000008 RSI: ffff8c314e369640 RDI: 00000000ffffffff
[    2.632071] RBP: ffff8c3155914828 R08: ffffffffb09cfac0 R09: 0000000000000001
[    2.632090] R10: 0000000000000019 R11: ffffffffb09cfb50 R12: ffff8c3155935e00
[    2.632109] R13: ffff8c3155914828 R14: ffff8c3155914960 R15: ffff8c31559148a4
[    2.632129] FS:  000072ea2b67fb80(0000) GS:ffff8c3156f00000(0000) knlGS:0000000000000000
[    2.632149] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.632166] CR2: ffffffffff5f9000 CR3: 000000000c266005 CR4: 00000000001606e0
[    2.632185] Kernel panic - not syncing: Fatal exception
[    2.632319] Kernel Offset: 0x2e000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Jürgen Groß]

On 27.09.20 13:14, Marek Marczykowski-Górecki wrote:
> Hi all,
> 
> I get kernel panic on 'floppy' module load. If I blacklist the module,
> then everything works.
> The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
> fine. PV dom0 works too. I haven't tried bare metal, but I assume it
> works fine too.

Could you please try bare metal?

Working in PV and PVH mode, but not in HVM, is pointing towards either
an issue in IRQ handling (not Xen specific) or in qemu.

It might be that Xen is advertising a rarely used APIC mode. In case
bare metal is working it might be helpful to have APIC related boot
message differences between HVM guest and bare metal.


Juergen

Re: Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On Mon, Sep 28, 2020 at 07:02:19AM +0200, Jürgen Groß wrote:
> On 27.09.20 13:14, Marek Marczykowski-Górecki wrote:
> > Hi all,
> > 
> > I get kernel panic on 'floppy' module load. If I blacklist the module,
> > then everything works.
> > The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
> > fine. PV dom0 works too. I haven't tried bare metal, but I assume it
> > works fine too.
> 
> Could you please try bare metal?

I don't have any hw with floppy controller at hand...
Booting on what I have, it works, loading floppy just says -ENODEV.

> Working in PV and PVH mode, but not in HVM, is pointing towards either
> an issue in IRQ handling (not Xen specific) or in qemu.
> 
> It might be that Xen is advertising a rarely used APIC mode. In case
> bare metal is working it might be helpful to have APIC related boot
> message differences between HVM guest and bare metal.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Denis Efremov]

Hi,

On 9/28/20 12:36 PM, Marek Marczykowski-Górecki wrote:
> On Mon, Sep 28, 2020 at 07:02:19AM +0200, Jürgen Groß wrote:
>> On 27.09.20 13:14, Marek Marczykowski-Górecki wrote:
>>> Hi all,
>>>
>>> I get kernel panic on 'floppy' module load. If I blacklist the module,
>>> then everything works.
>>> The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
>>> fine. PV dom0 works too. I haven't tried bare metal, but I assume it
>>> works fine too.
>>
>> Could you please try bare metal?
> 
> I don't have any hw with floppy controller at hand...
> Booting on what I have, it works, loading floppy just says -ENODEV.

I saw that the issue was bisected [1] to commit
c330fb1ddc0a ("XEN uses irqdesc::irq_data_common::handler_data to store a
per interrupt XEN data pointer which contains XEN specific information.")

I have hardware, but I've never worked with Xen. It will take me some time
to set it up and reproduce the problem. I think I will do it in a week.

[1] https://github.com/QubesOS/qubes-issues/issues/6074#issuecomment-699636351


Thanks,
Denis

Re: Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

On Tue, Sep 29, 2020 at 03:27:43PM +0300, Denis Efremov wrote:
> Hi,
> 
> On 9/28/20 12:36 PM, Marek Marczykowski-Górecki wrote:
> > On Mon, Sep 28, 2020 at 07:02:19AM +0200, Jürgen Groß wrote:
> >> On 27.09.20 13:14, Marek Marczykowski-Górecki wrote:
> >>> Hi all,
> >>>
> >>> I get kernel panic on 'floppy' module load. If I blacklist the module,
> >>> then everything works.
> >>> The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
> >>> fine. PV dom0 works too. I haven't tried bare metal, but I assume it
> >>> works fine too.
> >>
> >> Could you please try bare metal?
> > 
> > I don't have any hw with floppy controller at hand...
> > Booting on what I have, it works, loading floppy just says -ENODEV.
> 
> I saw that the issue was bisected [1] to commit
> c330fb1ddc0a ("XEN uses irqdesc::irq_data_common::handler_data to store a
> per interrupt XEN data pointer which contains XEN specific information.")
> 
> I have hardware, but I've never worked with Xen. It will take me some time
> to set it up and reproduce the problem. I think I will do it in a week.

Can you try to boot 4.19.143 (or any other including that commit)
directly on the hardware and make sure floppy module is loaded? We do
know it's broken in Xen HVM, it would be interested to see if it works
without Xen. Even better if you could use the same kernel config:
https://gist.github.com/marmarek/1e6a359c9a99af3ed8fc16af0f36d8a6

> [1] https://github.com/QubesOS/qubes-issues/issues/6074#issuecomment-699636351

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Jürgen Groß]

[-- Attachment #1: Type: text/plain, Size: 1693 bytes --]

On 29.09.20 14:41, Marek Marczykowski-Górecki wrote:
> On Tue, Sep 29, 2020 at 03:27:43PM +0300, Denis Efremov wrote:
>> Hi,
>>
>> On 9/28/20 12:36 PM, Marek Marczykowski-Górecki wrote:
>>> On Mon, Sep 28, 2020 at 07:02:19AM +0200, Jürgen Groß wrote:
>>>> On 27.09.20 13:14, Marek Marczykowski-Górecki wrote:
>>>>> Hi all,
>>>>>
>>>>> I get kernel panic on 'floppy' module load. If I blacklist the module,
>>>>> then everything works.
>>>>> The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
>>>>> fine. PV dom0 works too. I haven't tried bare metal, but I assume it
>>>>> works fine too.
>>>>
>>>> Could you please try bare metal?
>>>
>>> I don't have any hw with floppy controller at hand...
>>> Booting on what I have, it works, loading floppy just says -ENODEV.
>>
>> I saw that the issue was bisected [1] to commit
>> c330fb1ddc0a ("XEN uses irqdesc::irq_data_common::handler_data to store a
>> per interrupt XEN data pointer which contains XEN specific information.")
>>
>> I have hardware, but I've never worked with Xen. It will take me some time
>> to set it up and reproduce the problem. I think I will do it in a week.
> 
> Can you try to boot 4.19.143 (or any other including that commit)
> directly on the hardware and make sure floppy module is loaded? We do
> know it's broken in Xen HVM, it would be interested to see if it works
> without Xen. Even better if you could use the same kernel config:
> https://gist.github.com/marmarek/1e6a359c9a99af3ed8fc16af0f36d8a6

I think it is not directly related to floppy, but a more general problem
for HVM guests.

I'm suspecting an issue with legacy IRQs. Could you please try the
attached patch?


Juergen

[-- Attachment #2: 0001-xen-events-don-t-use-chip_data-for-legacy-IRQs.patch --]
[-- Type: text/x-patch, Size: 3165 bytes --]

From 35b71ae9cc69cdd151cc3a4d587f67eb8d86007d Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
Date: Tue, 29 Sep 2020 15:47:21 +0200
Subject: [PATCH] xen/events: don't use chip_data for legacy IRQs

Storing Xen specific data via chip_data is fine, as long as this isn't
done for a legacy IRQ.

Use a local array for this purpose in case of legacy IRQs.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
 drivers/xen/events/events_base.c | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c
index 90b8f56fbadb..6f02c18fa65c 100644
--- a/drivers/xen/events/events_base.c
+++ b/drivers/xen/events/events_base.c
@@ -92,6 +92,8 @@ static bool (*pirq_needs_eoi)(unsigned irq);
 /* Xen will never allocate port zero for any purpose. */
 #define VALID_EVTCHN(chn)	((chn) != 0)
 
+static struct irq_info *legacy_info_ptrs[NR_IRQS_LEGACY];
+
 static struct irq_chip xen_dynamic_chip;
 static struct irq_chip xen_percpu_chip;
 static struct irq_chip xen_pirq_chip;
@@ -156,7 +158,18 @@ int get_evtchn_to_irq(evtchn_port_t evtchn)
 /* Get info for IRQ */
 struct irq_info *info_for_irq(unsigned irq)
 {
-	return irq_get_chip_data(irq);
+	if (irq < nr_legacy_irqs())
+		return legacy_info_ptrs[irq];
+	else
+		return irq_get_chip_data(irq);
+}
+
+static void set_info_for_irq(unsigned int irq, struct irq_info *info)
+{
+	if (irq < nr_legacy_irqs())
+		legacy_info_ptrs[irq] = info;
+	else
+		irq_set_chip_data(irq, info);
 }
 
 /* Constructors for packed IRQ information. */
@@ -377,7 +390,7 @@ static void xen_irq_init(unsigned irq)
 	info->type = IRQT_UNBOUND;
 	info->refcnt = -1;
 
-	irq_set_chip_data(irq, info);
+	set_info_for_irq(irq, info);
 
 	list_add_tail(&info->list, &xen_irq_list_head);
 }
@@ -426,14 +439,14 @@ static int __must_check xen_allocate_irq_gsi(unsigned gsi)
 
 static void xen_free_irq(unsigned irq)
 {
-	struct irq_info *info = irq_get_chip_data(irq);
+	struct irq_info *info = info_for_irq(irq);
 
 	if (WARN_ON(!info))
 		return;
 
 	list_del(&info->list);
 
-	irq_set_chip_data(irq, NULL);
+	set_info_for_irq(irq, NULL);
 
 	WARN_ON(info->refcnt > 0);
 
@@ -603,7 +616,7 @@ EXPORT_SYMBOL_GPL(xen_irq_from_gsi);
 static void __unbind_from_irq(unsigned int irq)
 {
 	evtchn_port_t evtchn = evtchn_from_irq(irq);
-	struct irq_info *info = irq_get_chip_data(irq);
+	struct irq_info *info = info_for_irq(irq);
 
 	if (info->refcnt > 0) {
 		info->refcnt--;
@@ -1108,7 +1121,7 @@ int bind_ipi_to_irqhandler(enum ipi_vector ipi,
 
 void unbind_from_irqhandler(unsigned int irq, void *dev_id)
 {
-	struct irq_info *info = irq_get_chip_data(irq);
+	struct irq_info *info = info_for_irq(irq);
 
 	if (WARN_ON(!info))
 		return;
@@ -1142,7 +1155,7 @@ int evtchn_make_refcounted(evtchn_port_t evtchn)
 	if (irq == -1)
 		return -ENOENT;
 
-	info = irq_get_chip_data(irq);
+	info = info_for_irq(irq);
 
 	if (!info)
 		return -ENOENT;
@@ -1170,7 +1183,7 @@ int evtchn_get(evtchn_port_t evtchn)
 	if (irq == -1)
 		goto done;
 
-	info = irq_get_chip_data(irq);
+	info = info_for_irq(irq);
 
 	if (!info)
 		goto done;
-- 
2.26.2

Re: Kernel panic on 'floppy' module load, Xen HVM, since 4.19.143

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2137 bytes --]

On Tue, Sep 29, 2020 at 04:05:21PM +0200, Jürgen Groß wrote:
> On 29.09.20 14:41, Marek Marczykowski-Górecki wrote:
> > On Tue, Sep 29, 2020 at 03:27:43PM +0300, Denis Efremov wrote:
> > > Hi,
> > > 
> > > On 9/28/20 12:36 PM, Marek Marczykowski-Górecki wrote:
> > > > On Mon, Sep 28, 2020 at 07:02:19AM +0200, Jürgen Groß wrote:
> > > > > On 27.09.20 13:14, Marek Marczykowski-Górecki wrote:
> > > > > > Hi all,
> > > > > > 
> > > > > > I get kernel panic on 'floppy' module load. If I blacklist the module,
> > > > > > then everything works.
> > > > > > The issue happens in Xen HVM, other virtualization modes (PV, PVH) works
> > > > > > fine. PV dom0 works too. I haven't tried bare metal, but I assume it
> > > > > > works fine too.
> > > > > 
> > > > > Could you please try bare metal?
> > > > 
> > > > I don't have any hw with floppy controller at hand...
> > > > Booting on what I have, it works, loading floppy just says -ENODEV.
> > > 
> > > I saw that the issue was bisected [1] to commit
> > > c330fb1ddc0a ("XEN uses irqdesc::irq_data_common::handler_data to store a
> > > per interrupt XEN data pointer which contains XEN specific information.")
> > > 
> > > I have hardware, but I've never worked with Xen. It will take me some time
> > > to set it up and reproduce the problem. I think I will do it in a week.
> > 
> > Can you try to boot 4.19.143 (or any other including that commit)
> > directly on the hardware and make sure floppy module is loaded? We do
> > know it's broken in Xen HVM, it would be interested to see if it works
> > without Xen. Even better if you could use the same kernel config:
> > https://gist.github.com/marmarek/1e6a359c9a99af3ed8fc16af0f36d8a6
> 
> I think it is not directly related to floppy, but a more general problem
> for HVM guests.
> 
> I'm suspecting an issue with legacy IRQs. Could you please try the
> attached patch?

Yes, this fixes the issue.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]