Re: xen/evtchn: Dom0 boot hangs using preempt_rt kernel 5.10

[Julien Grall <julien@xen.org>]

On 23/03/2021 17:06, Luca Fancellu wrote:
> Hi all,

Hi,

Please avoid top posting when answering to a comment. This makes more 
difficult to follow.

> I have an update, changing the lock introduced by the serie from spinlock_t to raw_spinlock_t, changing the lock/unlock function to use the raw_* version and keeping the BUG_ON(…) (now we can because raw_* implementation disable interrupts on preempt_rt) the kernel is booting correctly.
> So seems that the BUG_ON(…) is needed and the unmask function should run with interrupt disabled, anyone knows why this change worked?

Do you mean why no-one spotted the issue before? If so, AFAIK, on 
vanilla Linux, spin_lock is still just a wrapper to raw_spinlock. IOW 
there is no option to replace it with a RT spinlock.

So if you don't apply the RT patches, you would not be able to trigger 
the issue.

As to the fix itself, I think using raw_spinlock_t is the correct thing 
to do because the lock is also used in interrupt context (even with RT 
enabled).

Would you be able to send a patch?

> 
>> On 23 Mar 2021, at 15:39, Luca Fancellu <luca.fancellu@arm.com> wrote:
>>
>> Hi Jason,
>>
>> Thanks for your hints, unfortunately seems not an init problem because in the same init configuration I tried the 5.10.23 (preempt_rt) without the Juergen patch but with the BUG_ON removed and it boots without problem. So seems that applying the serie does something (on a preempt_rt kernel) and we are trying to figure out what.
>>
>>
>>> On 23 Mar 2021, at 12:36, Jason Andryuk <jandryuk@gmail.com> wrote:
>>>
>>> On Mon, Mar 22, 2021 at 3:09 PM Luca Fancellu <luca.fancellu@arm.com> wrote:
>>>>
>>>> Hi Juergen,
>>>>
>>>> Yes you are right it was my mistake, as you said to remove the BUG_ON(…) this serie (https://patchwork.kernel.org/project/xen-devel/cover/20210306161833.4552-1-jgross@suse.com/) is needed, since I’m using yocto I’m able to build a preempt_rt kernel up to the 5.10.23 and for this reason I’m applying that serie on top of this version, then I’m removing the BUG_ON(…).
>>>>
>>>> A thing that was not expected is that now the Dom0 kernel is stuck on “Setting domain 0 name, domid and JSON config…” step and the system seems unresponsive. Seems like a deadlock issue but looking into the serie we can’t spot anything and that serie was also tested by others from the community.

The deadlock is expected. When you enable RT spinlock, the interrupts 
will not disabled even when you call spin_lock_irqsave().

As the lock is also used in interrupt context (e.g. with interrupt 
masked), this will lead to a deadlock because the lock can be held with 
interrupt unmasked.

This is quite a common error as developpers are not yet used to test RT. 
I remember finding a few other instances like that when I worked on RT a 
couple of years ago.

For future reference, I think CONFIG_PROVE_LOCKING=y could help you to 
detect (potential) deadlock.

Cheers,

-- 
Julien Grall