Re: System Hang With 5.15.79-rt54 Patch Set

[Joseph Salisbury <joseph.salisbury@canonical.com>]

On 12/15/22 18:39, Joseph Salisbury wrote:
>
>
> On 12/15/22 13:39, Joseph Salisbury wrote:
>> Hello,
>>
>> A bug has been reported[0] against a linux-5.15.y based Ubuntu kernel 
>> that has the 5.15.79-rt54 real-time patch set applied.
>>
>> This bug is causing a system hang when running disk tests with the 
>> stress-ng[1] test suite.  A 'hung task' call trace is being 
>> generated[2] just prior to the system hanging.  It appears the hang 
>> is happening during a call to schedule.
>>
>> To see if this was a regression, I also tested prior real-time patch 
>> sets, going as far back to rt38 and all versions exhibit this bug. I 
>> will test back further to see if this is in fact a regression and a 
>> kernel bisect can be done.  I also confirmed the same kernel without 
>> the rt54 patch set applied does not exhibit this bug.
>>
>> I will continue to debug this issue, but I wanted to send this 
>> message upstream in case anyone else runs into it.  Any specific 
>> tests, tracing or debugging steps would be greatly appreciated. 
>> Otherwise, I'll report back to this thread with any findings.
>>
>> Thanks,
>>
>> Joe
>>
>>
>> [0] https://bugs.launchpad.net/ubuntu-realtime/+bug/1998536
>> [1] https://github.com/ColinIanKing/stress-ng
>> [2] https://launchpadlibrarian.net/637058129/dmesg_dl110_rt_kernel.txt
>
> I should have also mentioned, I will also test the latest available 
> patch set (6.1-rc7-rt5) to see if it also exhibits the bug.
>
>
This bug/system hang does still happen with the v6.1-rc7 kernel and the 
rt5 real-time patch.  I also confirmed the bug/system hang does not 
happen with just v6.1-rc7 without the real-time patch applied.

I will bisect through the rt patches to see if I can find the specific 
patch(s) that causes the bug.  I will use the v5.15 kernel and it's rt 
patches, since v5.15 has the patches broken out individually, and they 
can be applied one at a time with 'git am'.

I'll provide further updates during the bisect.

Thanks,

Joe