PCI probe fails (Cobalt 3.2.2 / Dovetail 5.15.y-dovetail)

PCI probe fails (Cobalt 3.2.2 / Dovetail 5.15.y-dovetail)

[Auel, Kendall]

I'm porting a working PCI device driver from Cobalt/IPipe to Cobalt/Dovetail and everything builds normally. When I attempt to install the driver I get the error pasted below. I tried sending the IRQF_SHARED flag in rtdm_irq_request but it didn't help (the flag was ignored perhaps?). Any thoughts on what to try next? I suppose I'll trace through the cobalt code and into dovetail to see why the shared flag didn't take and why xnintr_destroy is being called.

Aug 26 11:21:26 xeno-mx3110h-01 kernel: genirq: Flags mismatch irq 16. 00210000 (bolt) vs. 00000080 (i801_smbus) 
Aug 26 11:21:26 xeno-mx3110h-01 kernel: ------------[ cut here ]------------
Aug 26 11:21:26 xeno-mx3110h-01 kernel: Trying to free already-free IRQ 16
Aug 26 11:21:26 xeno-mx3110h-01 kernel: WARNING: CPU: 3 PID: 47441 at kernel/irq/manage.c:1942 free_irq+0x249/0x470
Aug 26 11:21:26 xeno-mx3110h-01 kernel: Modules linked in: bolt_driver(OE+) evbus_driver(OE) nls_iso8859_1 i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_>
Aug 26 11:21:26 xeno-mx3110h-01 kernel: CPU: 3 PID: 47441 Comm: insmod Tainted: G        W  OE     5.15.57-bolt #1
Aug 26 11:21:26 xeno-mx3110h-01 kernel: Hardware name: Default string Default string/Default string, BIOS 1.01B 09/21/2020
Aug 26 11:21:26 xeno-mx3110h-01 kernel: IRQ stage: Linux
Aug 26 11:21:26 xeno-mx3110h-01 kernel: RIP: 0010:free_irq+0x249/0x470
Aug 26 11:21:26 xeno-mx3110h-01 kernel: Code: 41 0f b1 94 24 c0 00 00 00 0f 85 c0 01 00 00 9c 41 59 e9 76 fe ff ff 44 89 fe 48 c7 c7 18 d2 b2 82 4c 89 4d c8 e8 a3 58 9f 00 >
Aug 26 11:21:26 xeno-mx3110h-01 kernel: RSP: 0018:ffffa52ec15579b0 EFLAGS: 00010082
Aug 26 11:21:26 xeno-mx3110h-01 kernel: RAX: 0000000000000000 RBX: ffffffffc0a23aa0 RCX: 0000000000000027
Aug 26 11:21:26 xeno-mx3110h-01 kernel: RDX: 0000000000000027 RSI: 80000000ffffe62f RDI: ffff8ac67659f4c8
Aug 26 11:21:26 xeno-mx3110h-01 kernel: RBP: ffffa52ec15579e8 R08: ffff8ac67659f4c0 R09: 0000000000000001
Aug 26 11:21:26 xeno-mx3110h-01 kernel: R10: 0000000000000001 R11: ffffffff8335689a R12: ffff8ac5001f6000
Aug 26 11:21:26 xeno-mx3110h-01 kernel: R13: ffff8ac5001f6180 R14: ffff8ac500cfec80 R15: 0000000000000010
Aug 26 11:21:26 xeno-mx3110h-01 kernel: FS:  00007f1174eda540(0000) GS:ffff8ac676580000(0000) knlGS:0000000000000000
Aug 26 11:21:26 xeno-mx3110h-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 26 11:21:26 xeno-mx3110h-01 kernel: CR2: 000055ba0acd3c28 CR3: 00000001025ec002 CR4: 00000000003706e0
Aug 26 11:21:26 xeno-mx3110h-01 kernel: Call Trace:
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  <TASK>
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  xnintr_detach+0x25/0x30
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  xnintr_destroy+0xe/0x20
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  rtdm_irq_request_affine+0x82/0xa0
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  rtdm_irq_request+0x10/0x20
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  bolt_pci_probe+0x192/0x240 [bolt_driver]
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  local_pci_probe+0x48/0x90
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  pci_device_probe+0x113/0x1c0
Aug 26 11:21:26 xeno-mx3110h-01 kernel:  really_probe+0x1a4/0x3e0


Kendall Auel, Senior Embedded Software Engineer
3D Systems, 26600 SW Parkway Ave., Suite 300, Dock 61, Wilsonville OR 97070, USA
Tel: +1 503 482 1970   Mobile: +1 503 939 6862   http://3dsystems.com/

Re: PCI probe fails (Cobalt 3.2.2 / Dovetail 5.15.y-dovetail)

[Philippe Gerum]

Hi Kendall,

"Auel, Kendall" <Kendall.Auel@3DSystems.Com> writes:

> I'm porting a working PCI device driver from Cobalt/IPipe to
> Cobalt/Dovetail and everything builds normally. When I attempt to
> install the driver I get the error pasted below. I tried sending the
> IRQF_SHARED flag in rtdm_irq_request but it didn't help (the flag was
> ignored perhaps?). Any thoughts on what to try next? I suppose I'll
> trace through the cobalt code and into dovetail to see why the shared
> flag didn't take and why xnintr_destroy is being called.
>
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: genirq: Flags mismatch irq 16. 00210000 (bolt) vs. 00000080 (i801_smbus) 
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: ------------[ cut here ]------------
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: Trying to free already-free IRQ 16
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: WARNING: CPU: 3 PID: 47441 at kernel/irq/manage.c:1942 free_irq+0x249/0x470
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: Modules linked in: bolt_driver(OE+) evbus_driver(OE) nls_iso8859_1 i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_>
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: CPU: 3 PID: 47441 Comm: insmod Tainted: G        W  OE     5.15.57-bolt #1
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: Hardware name: Default string Default string/Default string, BIOS 1.01B 09/21/2020
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: IRQ stage: Linux
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: RIP: 0010:free_irq+0x249/0x470
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: Code: 41 0f b1 94 24 c0 00 00 00 0f 85 c0 01 00 00 9c 41 59 e9 76 fe ff ff 44 89 fe 48 c7 c7 18 d2 b2 82 4c 89 4d c8 e8 a3 58 9f 00 >
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: RSP: 0018:ffffa52ec15579b0 EFLAGS: 00010082
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: RAX: 0000000000000000 RBX: ffffffffc0a23aa0 RCX: 0000000000000027
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: RDX: 0000000000000027 RSI: 80000000ffffe62f RDI: ffff8ac67659f4c8
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: RBP: ffffa52ec15579e8 R08: ffff8ac67659f4c0 R09: 0000000000000001
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: R10: 0000000000000001 R11: ffffffff8335689a R12: ffff8ac5001f6000
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: R13: ffff8ac5001f6180 R14: ffff8ac500cfec80 R15: 0000000000000010
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: FS:  00007f1174eda540(0000) GS:ffff8ac676580000(0000) knlGS:0000000000000000
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: CR2: 000055ba0acd3c28 CR3: 00000001025ec002 CR4: 00000000003706e0
> Aug 26 11:21:26 xeno-mx3110h-01 kernel: Call Trace:
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  <TASK>
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  xnintr_detach+0x25/0x30
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  xnintr_destroy+0xe/0x20
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  rtdm_irq_request_affine+0x82/0xa0
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  rtdm_irq_request+0x10/0x20
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  bolt_pci_probe+0x192/0x240 [bolt_driver]
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  local_pci_probe+0x48/0x90
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  pci_device_probe+0x113/0x1c0
> Aug 26 11:21:26 xeno-mx3110h-01 kernel:  really_probe+0x1a4/0x3e0
>
>

There are multiple issues:

- Dovetail does allow sharing lines for oob interrupts [1], _only_ if
  IRQF_OOB|IRQF_SHARED are set for all handlers attached to the shared
  IRQ line. "genirq: Flags mismatch" reports an issue with this. The
  smbus handler was set to IRQF_SHARED (0x00000080), whilst the 'bolt'
  IRQ handler was given IRQF_OOB[|IRQF_NO_THREAD]. In order to share
  this line with the 'bolt' device, you would need to enable IRQF_OOB on
  the smbus handler as well.

- Now, enabling IRQF_SHARED on an oob line is not possible with the
  current implementation of xnintr_attach() in 3.2.2, which forces the
  irqflags to IRQF_OOB. At the very least, you would need a way to pass
  IRQF_OOB|IRQF_SHARED there.

- xnintr_destroy() assumes the IRQ was successfully attached/requested,
  as it attempts to free it. Cobalt/I-pipe had a state bit guarding this
  condition, which does not exist in the Dovetail-based version of
  Cobalt. As a result, rtdm_irq_request_affine() calls xnintr_destroy()
  unconditionally to cleanup the mess after xnintr_attach() failed,
  causing this report: "Trying to free already-free IRQ 16".

Do you actually have to share IRQ16 between the bolt and smbus devices?
Although possible via some ugly hack, sharing IRQs between rt and non-rt
devices is a real problem.

[1]
https://evlproject.org/dovetail/pipeline/irq_handling/#request-oob-irq

-- 
Philippe.

RE: PCI probe fails (Cobalt 3.2.2 / Dovetail 5.15.y-dovetail)

[Auel, Kendall]

Hi Philippe,

> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Saturday, August 27, 2022 1:14 AM
> 
> Hi Kendall,
> 
> There are multiple issues:
> 
> - Dovetail does allow sharing lines for oob interrupts [1], _only_ if
>   IRQF_OOB|IRQF_SHARED are set for all handlers attached to the shared
>   IRQ line. "genirq: Flags mismatch" reports an issue with this. The
>   smbus handler was set to IRQF_SHARED (0x00000080), whilst the 'bolt'
>   IRQ handler was given IRQF_OOB[|IRQF_NO_THREAD]. In order to share
>   this line with the 'bolt' device, you would need to enable IRQF_OOB on
>   the smbus handler as well.
> 
> - Now, enabling IRQF_SHARED on an oob line is not possible with the
>   current implementation of xnintr_attach() in 3.2.2, which forces the
>   irqflags to IRQF_OOB. At the very least, you would need a way to pass
>   IRQF_OOB|IRQF_SHARED there.
> 
> - xnintr_destroy() assumes the IRQ was successfully attached/requested,
>   as it attempts to free it. Cobalt/I-pipe had a state bit guarding this
>   condition, which does not exist in the Dovetail-based version of
>   Cobalt. As a result, rtdm_irq_request_affine() calls xnintr_destroy()
>   unconditionally to cleanup the mess after xnintr_attach() failed,
>   causing this report: "Trying to free already-free IRQ 16".
> 
> Do you actually have to share IRQ16 between the bolt and smbus devices?
> Although possible via some ugly hack, sharing IRQs between rt and non-rt
> devices is a real problem.

I changed the kernel config from "CONFIG_I2C_I801=m" to
"CONFIG_I2C_SMBUS=m" and the issue went away. There do not
seem to be any negative side effects for my system.

The I801 device is working okay alongside the RTDM driver and
IPipe (patched kernel version 5.4.180). So perhaps this is only a
dovetail limitation, but in any case I'm able to move forward
with the 5.15 kernel.

Regards,
Kendall