fixmap problem on PA11 hardware

fixmap problem on PA11 hardware

[Helge Deller]

When booting the same binary 32-bit kernel (via network),
it boots sucessfully on PA7300 and above CPUs, while it
fails with DTLB fault (trap #15) on my PA7100LC (PA11) machine.

I've narrowed the problem down to the fixmap implementation.
Here is the relevant part of the boot log (with some debug info included):

Decompressing Linux... done.
Booting the kernel.
Linux version 5.15.0-rc6-32bit+ (deller@p100) (hppa-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.34-2.fc33) #836 PREEMPT Wed Oct 27 20:01:14 CEST 2021
FP[0] enabled: Rev 1 Model 13
The 32-bit Kernel has started...
...
TOC handler registered
rcu: Hierarchical SRCU implementation.
patch_multiple 101856d8  len = 4
set_fixmap  phys = 0x185000   virt = 0xeffe000    parisc_tlb_flush_threshold = 0xffffffff  FIXMAP_START = 0xeffe000
patch_multiple map at 0effe6d8
TRAP 15  0xeffe6d8
Backtrace:
 [<107bff04>] patch_text+0x28/0x40
 [<107c0358>] arch_arm_kprobe+0x20/0x30
 [<1022be4c>] arm_kprobe+0x4c/0x7c
 [<1022e1c0>] register_kprobe.part.0+0x150/0x16c
 [<1022e228>] register_kprobe+0x4c/0x60
 [<1010858c>] 0x1010858c
 [<1010f178>] 0x1010f178
 [<1017711c>] do_one_initcall+0x80/0x1ac
 [<10103378>] 0x10103378
 [<107b98e4>] kernel_init+0x2c/0x154
 [<1017401c>] ret_from_kernel_thread+0x1c/0x24


This is a log from the PA7300LC CPU:

rcu: Hierarchical SRCU implementation.
patch_multiple 101856d8  len = 4
set_fixmap  phys = 0x185000   virt = 0xeffe000    parisc_tlb_flush_threshold = 0xffffffff  FIXMAP_START = 0xeffe000
patch_multiple map at 0effe6d8
Kprobe smoke test: started
patch_multiple 1021e800  len = 4
set_fixmap  phys = 0x21e000   virt = 0xeffe000    parisc_tlb_flush_threshold = 0xffffffff  FIXMAP_START = 0xeffe000
patch_multiple map at 0effe800
...

Again, this is exactly the same kernel booted on different machines.

The only explanation I have for this is, that I believe the dtlb
fault handler includes some assembler statements which are executed
differently on PA7100 vs. PA7300 (and higher) CPUs.

In the archives I found this old thread, which might be related:
https://www.spinics.net/lists/linux-parisc/msg09391.html

Anyway, here is the assembly:

10175738 <dtlb_miss_11>:
10175738:       23 3f a0 14     ldil L%abf000,r25
1017573c:       37 39 00 00     ldo 0(r25),r25
10175740:       0b 00 22 40     or,= r0,r24,r0
10175744:       03 20 08 b9     mfctl tr1,r25
10175748:       00 00 e4 a1     mfsp sr7,r1
1017574c:       0b 00 32 40     or,<> r0,r24,r0
10175750:       08 18 02 41     copy r24,r1
10175754:       08 20 22 40     or,= r0,r1,r0
10175758:       8b 01 31 72     cmpb,<>,n r1,r24,10176018 <dtlb_fault>
1017575c:       d1 01 19 36     extrw,u r8,9,10,r1
10175760:       d7 20 0c 14     depw r0,31,12,r25
10175764:       08 00 02 50     copy r0,r16
10175768:       0f 21 20 99     ldw,s r1(r25),r25
1017576c:       c7 f9 c0 e2     bb,>=,n r25,1f,101757e4 <dtlb_check_alias_11>
10175770:       d7 20 0c 1c     depw r0,31,4,r25
10175774:       08 19 02 49     copy r25,r9
10175778:       d7 29 09 08     depw,z r9,23,24,r25
1017577c:       d1 01 1a 76     extrw,u r8,19,10,r1
10175780:       d7 20 0c 14     depw r0,31,12,r25
10175784:       0b 21 06 99     shladd r1,2,r25,r25
10175788:       0f 20 10 90     ldw 0(r25),r16
1017578c:       c6 d0 c0 a2     bb,>=,n r16,16,101757e4 <dtlb_check_alias_11>
10175790:       34 09 02 00     ldi 100,r9
10175794:       0a 09 02 41     or r9,r16,r1
10175798:       0a 09 32 00     and,<> r9,r16,r0
1017579c:       0f 21 12 80     stw r1,0(r25)
101757a0:       d6 38 08 31     depw,z r24,30,15,r17
101757a4:       d6 30 0e f9     depw r16,8,7,r17
101757a8:       d2 00 3b 1f     extrw,u,= r16,24,1,r0
101757ac:       d6 22 1e 7f     depwi 1,12,1,r17
101757b0:       d2 00 3a 9f     extrw,u,= r16,20,1,r0
101757b4:       d6 2e 1e 9d     depwi 7,11,3,r17
101757b8:       d2 00 3b 9f     extrw,u,= r16,28,1,r0
101757bc:       d6 20 1e 9e     depwi 0,11,2,r17
101757c0:       d6 00 1c 14     depwi 0,31,12,r16
101757c4:       d2 10 1b 07     extrw,u r16,24,25,r16
101757c8:       00 00 44 a9     mfsp sr1,r9
101757cc:       00 18 58 20     mtsp r24,sr1
101757d0:       05 10 50 40     idtlba r16,(sr1,r8)
101757d4:       05 11 50 00     idtlbp r17,(sr1,r8)
101757d8:       00 09 58 20     mtsp r9,sr1
101757dc:       00 00 0c a0     rfi,r
101757e0:       08 00 02 40     nop

101757e4 <dtlb_check_alias_11>:
101757e4:       8f 00 30 5a     cmpib,<>,n 0,r24,10176018 <dtlb_fault>
101757e8:       20 20 01 e0     ldil L%f000000,r1
101757ec:       08 08 02 49     copy r8,r9
101757f0:       d5 20 1c 09     depwi 0,31,23,r9
101757f4:       89 21 30 3a     cmpb,<>,n r1,r9,10176018 <dtlb_fault>
101757f8:       02 60 08 a1     mfctl iir,r1
101757fc:       d0 21 18 ba     extrw,u r1,5,6,r1
10175800:       34 11 00 86     ldi 43,r17
10175804:       90 20 20 02     cmpiclr,= 1,r1,r0
10175808:       34 11 00 46     ldi 23,r17
1017580c:       d6 31 0a f9     depw,z r17,8,7,r17
10175810:       d1 00 39 3f     extrw,u,= r8,9,1,r0
10175814:       08 17 12 50     or,tr r23,r0,r16
10175818:       08 1a 02 50     copy r26,r16
1017581c:       05 10 10 40     idtlba r16,(r8)
10175820:       05 11 10 00     idtlbp r17,(r8)
10175824:       00 00 0c a0     rfi,r
10175828:       08 00 02 40     nop

Any idea what's going wrong or any suggestion I can try?
Btw, I did tested older kernels too, but they all showed the same problem.

Helge

Re: fixmap problem on PA11 hardware

[John David Anglin]

On 2021-10-27 3:09 p.m., Helge Deller wrote:
> In the archives I found this old thread, which might be related:
> https://www.spinics.net/lists/linux-parisc/msg09391.html
These seems unlikely as both 7100LC and 7300LC are PA-RISC 1.1 processors (1.1c vs. 1.1e).  Big difference
seems to be cache.

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: fixmap problem on PA11 hardware

[Helge Deller]

On 10/27/21 22:14, John David Anglin wrote:
> On 2021-10-27 3:09 p.m., Helge Deller wrote:
>> In the archives I found this old thread, which might be related:
>> https://www.spinics.net/lists/linux-parisc/msg09391.html
> These seems unlikely as both 7100LC and 7300LC are PA-RISC 1.1 processors (1.1c vs. 1.1e).  Big difference
> seems to be cache.

Yes, there were at least two problems.
I just sent two patches to the list which fix the crashes.
But FTRACE still doesn't work on PA1.x machines as expected,
while the same code seems to work on PA2.x machines (running the same 32bit kernel).

The PA7100LC reports (machine: 715/64):
Kprobe smoke test: started
Kprobe smoke test: kretprobe handler not called
Kprobe smoke test: kretprobe handler not called
Kprobe smoke test: kretprobe handler2 not called
Kprobe smoke test: BUG: 3 error(s) running handlers

On the PA7300LC (machine B160L):
Kprobe smoke test: started
Kprobe smoke test: kprobe pre_handler not called
Kprobe smoke test: kprobe post_handler not called
Kprobe smoke test: kprobe pre_handler not called
Kprobe smoke test: kprobe post_handler not called
Kprobe smoke test: kprobe pre_handler2 not called
Kprobe smoke test: kprobe post_handler2 not called
Kprobe smoke test: kretprobe handler not called
Kprobe smoke test: kretprobe handler not called
Kprobe smoke test: kretprobe handler2 not called
Kprobe smoke test: BUG: 9 error(s) running handlers

On the PA8x00 (machine C3000):
Kprobe smoke test: started
Kprobe smoke test: passed successfully

Helge

Re: fixmap problem on PA11 hardware

[John David Anglin]

Hi Helge,

My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.

Note also that flush_kernel_vmap_range and invalidate_kernel_vmap_range both have this hunk
which may behave differently:

         if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) &&
             (unsigned long)size >= parisc_cache_flush_threshold) {
                 flush_tlb_kernel_range(start, end);
                 flush_data_cache();
                 return;
         }

Dave

On 2021-10-31 5:22 p.m., Helge Deller wrote:
> On 10/27/21 22:14, John David Anglin wrote:
>> On 2021-10-27 3:09 p.m., Helge Deller wrote:
>>> In the archives I found this old thread, which might be related:
>>> https://www.spinics.net/lists/linux-parisc/msg09391.html
>> These seems unlikely as both 7100LC and 7300LC are PA-RISC 1.1 processors (1.1c vs. 1.1e).  Big difference
>> seems to be cache.
> Yes, there were at least two problems.
> I just sent two patches to the list which fix the crashes.
> But FTRACE still doesn't work on PA1.x machines as expected,
> while the same code seems to work on PA2.x machines (running the same 32bit kernel).


-- 
John David Anglin  dave.anglin@bell.net

Re: fixmap problem on PA11 hardware

[Helge Deller]

Hi Dave,

On 10/31/21 23:47, John David Anglin wrote:
> My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
> it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.

That's true, nevertheless I've seen different behaviour on real hardware.

It might be, that:

-	flush_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
+	invalidate_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
(here the flush might be sufficient)
 	if (mapped)
 		patch_unmap(FIX_TEXT_POKE0, &flags);
+	invalidate_kernel_vmap_range((void *)start, end - start);
(but here the pdc is necessary to actually drop data from I-cache)
 	flush_icache_range(start, end);

I can test tomorrow...

> Note also that flush_kernel_vmap_range and invalidate_kernel_vmap_range both have this hunk
> which may behave differently:
>
>         if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) &&
>             (unsigned long)size >= parisc_cache_flush_threshold) {
>                 flush_tlb_kernel_range(start, end);
>                 flush_data_cache();
>                 return;
>         }

Yes, but even when patching 4 bytes the patch above did worked (and
parisc_cache_flush_threshold was bigger).

Helge

>
> Dave
>
> On 2021-10-31 5:22 p.m., Helge Deller wrote:
>> On 10/27/21 22:14, John David Anglin wrote:
>>> On 2021-10-27 3:09 p.m., Helge Deller wrote:
>>>> In the archives I found this old thread, which might be related:
>>>> https://www.spinics.net/lists/linux-parisc/msg09391.html
>>> These seems unlikely as both 7100LC and 7300LC are PA-RISC 1.1 processors (1.1c vs. 1.1e).  Big difference
>>> seems to be cache.
>> Yes, there were at least two problems.
>> I just sent two patches to the list which fix the crashes.
>> But FTRACE still doesn't work on PA1.x machines as expected,
>> while the same code seems to work on PA2.x machines (running the same 32bit kernel).
>
>

Re: fixmap problem on PA11 hardware

[John David Anglin]

On 2021-10-31 7:07 p.m., Helge Deller wrote:
> On 10/31/21 23:47, John David Anglin wrote:
>> My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
>> it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.
> That's true, nevertheless I've seen different behaviour on real hardware.
>
> It might be, that:
>
> -	flush_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
> +	invalidate_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
> (here the flush might be sufficient)
>   	if (mapped)
>   		patch_unmap(FIX_TEXT_POKE0, &flags);
> +	invalidate_kernel_vmap_range((void *)start, end - start);
> (but here the pdc is necessary to actually drop data from I-cache)
>   	flush_icache_range(start, end);
>
> I can test tomorrow...
Maybe sync is needed before releasing lock.  pdc/fdc are weakly ordered.

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: fixmap problem on PA11 hardware

[John David Anglin]

On 2021-10-31 7:56 p.m., John David Anglin wrote:
> n 2021-10-31 7:07 p.m., Helge Deller wrote:
>> On 10/31/21 23:47, John David Anglin wrote:
>>> My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
>>> it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.
>> That's true, nevertheless I've seen different behaviour on real hardware.
>>
>> It might be, that:
>>
>> -    flush_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>> +    invalidate_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>> (here the flush might be sufficient)
>>       if (mapped)
>>           patch_unmap(FIX_TEXT_POKE0, &flags);
>> +    invalidate_kernel_vmap_range((void *)start, end - start);
>> (but here the pdc is necessary to actually drop data from I-cache)
>>       flush_icache_range(start, end);
>>
>> I can test tomorrow...
> Maybe sync is needed before releasing lock.  pdc/fdc are weakly ordered.
Forget that.  There already is a sync.

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: fixmap problem on PA11 hardware

[John David Anglin]

On 2021-10-31 8:01 p.m., John David Anglin wrote:
> On 2021-10-31 7:56 p.m., John David Anglin wrote:
>> n 2021-10-31 7:07 p.m., Helge Deller wrote:
>>> On 10/31/21 23:47, John David Anglin wrote:
>>>> My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
>>>> it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.
>>> That's true, nevertheless I've seen different behaviour on real hardware.
>>>
>>> It might be, that:
>>>
>>> -    flush_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>>> +    invalidate_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>>> (here the flush might be sufficient)
>>>       if (mapped)
>>>           patch_unmap(FIX_TEXT_POKE0, &flags);
>>> +    invalidate_kernel_vmap_range((void *)start, end - start);
>>> (but here the pdc is necessary to actually drop data from I-cache)
>>>       flush_icache_range(start, end);
>>>
>>> I can test tomorrow...
>> Maybe sync is needed before releasing lock.  pdc/fdc are weakly ordered.
> Forget that.  There already is a sync.
Could it be we incorrectly change the flush/purge operations to nops?
89:     ALTERNATIVE(88b, 89b, ALT_COND_NO_DCACHE, INSN_NOP)

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: fixmap problem on PA11 hardware

[Helge Deller]

On 11/1/21 01:06, John David Anglin wrote:
> On 2021-10-31 8:01 p.m., John David Anglin wrote:
>> On 2021-10-31 7:56 p.m., John David Anglin wrote:
>>> n 2021-10-31 7:07 p.m., Helge Deller wrote:
>>>> On 10/31/21 23:47, John David Anglin wrote:
>>>>> My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
>>>>> it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.
>>>> That's true, nevertheless I've seen different behaviour on real hardware.
>>>>
>>>> It might be, that:
>>>>
>>>> -    flush_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>>>> +    invalidate_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>>>> (here the flush might be sufficient)
>>>>       if (mapped)
>>>>           patch_unmap(FIX_TEXT_POKE0, &flags);
>>>> +    invalidate_kernel_vmap_range((void *)start, end - start);
>>>> (but here the pdc is necessary to actually drop data from I-cache)
>>>>       flush_icache_range(start, end);
>>>>
>>>> I can test tomorrow...
>>> Maybe sync is needed before releasing lock.  pdc/fdc are weakly ordered.
>> Forget that.  There already is a sync.
> Could it be we incorrectly change the flush/purge operations to nops?
> 89:     ALTERNATIVE(88b, 89b, ALT_COND_NO_DCACHE, INSN_NOP)

No, can't be.
The alternative patching hasn't run yet when the kprobe testcases run.

Nevertheless, I don't fully understand it yet.
The same code works differently depending on the machine.

Helge

Re: fixmap problem on PA11 hardware

[John David Anglin]

On 2021-11-01 2:09 p.m., Helge Deller wrote:
> On 11/1/21 01:06, John David Anglin wrote:
>> On 2021-10-31 8:01 p.m., John David Anglin wrote:
>>> On 2021-10-31 7:56 p.m., John David Anglin wrote:
>>>> n 2021-10-31 7:07 p.m., Helge Deller wrote:
>>>>> On 10/31/21 23:47, John David Anglin wrote:
>>>>>> My sense is the invalidate patch isn't correct.  The main difference between pdc and fdc is that
>>>>>> it is optional whether pdc writes the cache line back to memory when it's dirty at priority 0.
>>>>> That's true, nevertheless I've seen different behaviour on real hardware.
>>>>>
>>>>> It might be, that:
>>>>>
>>>>> -    flush_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>>>>> +    invalidate_kernel_vmap_range((void *)fixmap, (p-fixmap) * sizeof(*p));
>>>>> (here the flush might be sufficient)
>>>>>        if (mapped)
>>>>>            patch_unmap(FIX_TEXT_POKE0, &flags);
>>>>> +    invalidate_kernel_vmap_range((void *)start, end - start);
>>>>> (but here the pdc is necessary to actually drop data from I-cache)
>>>>>        flush_icache_range(start, end);
>>>>>
>>>>> I can test tomorrow...
>>>> Maybe sync is needed before releasing lock.  pdc/fdc are weakly ordered.
>>> Forget that.  There already is a sync.
>> Could it be we incorrectly change the flush/purge operations to nops?
>> 89:     ALTERNATIVE(88b, 89b, ALT_COND_NO_DCACHE, INSN_NOP)
> No, can't be.
> The alternative patching hasn't run yet when the kprobe testcases run.
I was concerned about the flush vs. purge write back behavior, not the kprobe problem.

Dave

-- 
John David Anglin  dave.anglin@bell.net