[Bug 204819] New: KASAN still got problems loading some modules at boot

[Bug 204819] New: KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

            Bug ID: 204819
           Summary: KASAN still got problems loading some modules at boot
           Product: Platform Specific/Hardware
           Version: 2.5
    Kernel Version: 5.3-rc8
          Hardware: PPC-32
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: PPC-32
          Assignee: platform_ppc-32@kernel-bugs.osdl.org
          Reporter: erhard_f@mailbox.org
        Regression: No

Created attachment 284931
  --> https://bugzilla.kernel.org/attachment.cgi?id=284931&action=edit
dmesg (5.3-rc8, PowerMac G4 DP)

I tried finding a minimal kernel .config for bug #204479 to rule out other
influences.

What I found was, that KASAN shows different BUG: Unable to handle kernel data
access at 0x.......", depending on which module gets loaded/used first, like:

[...]
Sep 12 00:12:09 T600 kernel: BUG: Unable to handle kernel data access at
0xfe205150
Sep 12 00:12:09 T600 kernel: Faulting instruction address: 0xc00261d8
Sep 12 00:12:09 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Sep 12 00:12:09 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Sep 12 00:12:09 T600 kernel: Modules linked in: sungem_phy(+) usb_common
Sep 12 00:12:09 T600 kernel: CPU: 0 PID: 117 Comm: systemd-udevd Tainted: G    
   W         5.3.0-rc8 #3
Sep 12 00:12:09 T600 kernel: NIP:  c00261d8 LR: c02355c4 CTR: c0236c6c
Sep 12 00:12:09 T600 kernel: REGS: eb7d5ae0 TRAP: 0300   Tainted: G        W   
      (5.3.0-rc8)
Sep 12 00:12:09 T600 kernel: MSR:  00009032 <EE,ME,IR,DR,RI>  CR: 48228828 
XER: 20000000
Sep 12 00:12:09 T600 kernel: DAR: fe205150 DSISR: 0a000000 
                             GPR00: c0236cb0 eb7d5b98 ebbd2320 fe205150
00000000 00000007 c01083dc 00000003 
                             GPR08: e991025c 1e205150 00000000 f1013000
c0236c6c 007d6980 eb7d5eb8 eb7d5eb0 
                             GPR16: f10504c0 00000124 f102a880 f103d000
c010426c c144c480 00000011 eb9ef9a0 
                             GPR24: f102a8dc f102a8e0 f1013000 f102a810
0000002d 00000001 0000003c f1028a80 
Sep 12 00:12:09 T600 kernel: NIP [c00261d8] __memset+0x10/0xc0
Sep 12 00:12:09 T600 kernel: LR [c02355c4] kasan_unpoison_shadow+0x20/0x40
Sep 12 00:12:09 T600 kernel: Call Trace:
Sep 12 00:12:09 T600 kernel: [eb7d5b98] [ee802420] 0xee802420 (unreliable)
Sep 12 00:12:09 T600 kernel: [eb7d5ba8] [c0236cb0]
__asan_register_globals+0x44/0x68
Sep 12 00:12:09 T600 kernel: [eb7d5bc8] [c01083e8] do_init_module+0x134/0x2c4
Sep 12 00:12:09 T600 kernel: [eb7d5bf8] [c010b1b8] load_module+0x2b60/0x2d1c
Sep 12 00:12:09 T600 kernel: [eb7d5df8] [c010b5d8] sys_finit_module+0x100/0x140
Sep 12 00:12:09 T600 kernel: [eb7d5f38] [c001a274] ret_from_syscall+0x0/0x34
Sep 12 00:12:09 T600 kernel: --- interrupt: c00 at 0x24f2c4
                                 LR = 0x43e35c
Sep 12 00:12:09 T600 kernel: Instruction dump:
Sep 12 00:12:09 T600 kernel: 7c062396 7d4021d6 7d080214 7cca3050 90e30000
91030004 7cc33378 4e800020 
Sep 12 00:12:09 T600 kernel: 28050004 418000a0 5084442e 5084801e <90830000>
4d820020 70600003 7ca02a14 
Sep 12 00:12:09 T600 kernel: ---[ end trace 00769ab21374844d ]---
[...]
This changes depending on which modules I build for the kernel. So I suspect
KASAN to still not get the module loading quite right for all modules. At least
on my system (PowerMac G4 DP).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #1 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284933
  --> https://bugzilla.kernel.org/attachment.cgi?id=284933&action=edit
kernel .config (5.3-rc8, PowerMac G4 DP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

Christophe Leroy (christophe.leroy@c-s.fr) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |christophe.leroy@c-s.fr

--- Comment #2 from Christophe Leroy (christophe.leroy@c-s.fr) ---
Could you please provide the content of /sys/kernel/debug/kernel_page_tables
together with the dmesg, and also an 'objdump -h' of the failing modules ?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

Erhard F. (erhard_f@mailbox.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #284931|0                           |1
        is obsolete|                            |

--- Comment #3 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284949
  --> https://bugzilla.kernel.org/attachment.cgi?id=284949&action=edit
dmesg (5.3-rc8, PowerMac G4 DP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #4 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284951
  --> https://bugzilla.kernel.org/attachment.cgi?id=284951&action=edit
kernel_page_tables (5.3-rc8, PowerMac G4 DP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #5 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284953
  --> https://bugzilla.kernel.org/attachment.cgi?id=284953&action=edit
objdump sungem (5.3-rc8, PowerMac G4 DP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #6 from Christophe Leroy (christophe.leroy@c-s.fr) ---
Aaarrgghhh ! kernel_page_tables addresses are wrong. Ok, I'll manage.

For next time, please apply
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/arch/powerpc/mm/ptdump?h=next-20190904&id=7c7a532ba3fc51bf9527d191fb410786c1fdc73c

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #7 from Christophe Leroy (christophe.leroy@c-s.fr) ---
Created attachment 284961
  --> https://bugzilla.kernel.org/attachment.cgi?id=284961&action=edit
Manually fixed kernel_page_tables

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #8 from Christophe Leroy (christophe.leroy@c-s.fr) ---
You get:

Sep 13 17:43:49 T600 kernel: BUG: Unable to handle kernel data access at
0xfe205150

However this area is properly mapped RW:

0xfe205000-0xfe205fff  0x297d6000         4K        rw       present          
dirty  accessed                         

There must be something wrong with hash table update.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #9 from Erhard F. (erhard_f@mailbox.org) ---
I can apply the patch but how do I apply the manually fixed kernel_page_tables?
Not quite sure how to proceed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #10 from Christophe Leroy (christophe.leroy@c-s.fr) ---
No no, what I mean by 'manually fixed kernel_page_tables' is that I fixed the
file you attached so that it has the same values as what it would have had with
the patch applied. Don't worry.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #11 from Christophe Leroy (christophe.leroy@c-s.fr) ---
I have carefully reviewed the flushing calls and have not been able to identify
any issue.

Maybe a SMP issue ? Does this problem also happen without CONFIG_SMP ?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #12 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284991
  --> https://bugzilla.kernel.org/attachment.cgi?id=284991&action=edit
dmesg v2 (5.3-rc8 + ptdump patch)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #13 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284993
  --> https://bugzilla.kernel.org/attachment.cgi?id=284993&action=edit
kernel_page_tables v2 (5.3-rc8 + ptdump patch)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #14 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284995
  --> https://bugzilla.kernel.org/attachment.cgi?id=284995&action=edit
objdump firewire-ohci v2 (5.3-rc8 + ptdump patch)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #15 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284997
  --> https://bugzilla.kernel.org/attachment.cgi?id=284997&action=edit
objdump usbcore v2 (5.3-rc8 + ptdump patch)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #16 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 284999
  --> https://bugzilla.kernel.org/attachment.cgi?id=284999&action=edit
dmesg v3 (5.3-rc8 + ptdump patch, NO SMP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #17 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 285001
  --> https://bugzilla.kernel.org/attachment.cgi?id=285001&action=edit
kernel_page_tables v3 (5.3-rc8 + ptdump patch, NO SMP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #18 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 285003
  --> https://bugzilla.kernel.org/attachment.cgi?id=285003&action=edit
objdump usbcore v3 (5.3-rc8 + ptdump patch, NO SMP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #19 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 285005
  --> https://bugzilla.kernel.org/attachment.cgi?id=285005&action=edit
objdump ohci-hcd v3 (5.3-rc8 + ptdump patch, NO SMP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #20 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 285007
  --> https://bugzilla.kernel.org/attachment.cgi?id=285007&action=edit
objdump firewire-ohci v3 (5.3-rc8 + ptdump patch, NO SMP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #21 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 285009
  --> https://bugzilla.kernel.org/attachment.cgi?id=285009&action=edit
objdump ehci-pci v3 (5.3-rc8 + ptdump patch, NO SMP)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #22 from Erhard F. (erhard_f@mailbox.org) ---
(In reply to Christophe Leroy from comment #11)
> I have carefully reviewed the flushing calls and have not been able to
> identify any issue.
> 
> Maybe a SMP issue ? Does this problem also happen without CONFIG_SMP ?
For my 1st (v2) retest I applied the "powerpc/ptdump: Fix addresses display on
PPC32" patch you linked on top of -rc8.

For my 2nd (v3) retest I applied the patch and disabled SMP in the .config.

As you can see the issue still persists without SMP. Also different
'problematic' modules popup even without SMP.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #23 from Christophe Leroy (christophe.leroy@c-s.fr) ---
Thanks. The third one is interesting, as it shows :

Sep 16 12:57:39 T600 kernel: BUG: Unable to handle kernel data access at
0xfe295404

0xfe295000-0xfe295fff  0x00d05000         4K  user  r        present           
      accessed                         


So unlike the other cases, this one shows that the area is still read-only and
pointing to the zero shadow area.
Wondering whether for the other ones the area is not allocated to a further
module loader after the failing one.

Are you able to dump kernel_page_tables just after each module load failure
before loading any additional module ?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

--- Comment #24 from Christophe Leroy (christophe.leroy@c-s.fr) ---
Do you confirm that the two patches done in the scope of bug #204479 are
included in your config, especially the first one:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/arch/powerpc/mm/kasan/kasan_init_32.c?id=663c0c9496a69f80011205ba3194049bcafd681d
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/arch/powerpc/mm/kasan/kasan_init_32.c?id=45ff3c55958542c3b76075d59741297b8cb31cbb

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

Erhard F. (erhard_f@mailbox.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #25 from Erhard F. (erhard_f@mailbox.org) ---
(In reply to Christophe Leroy from comment #24)
> Do you confirm that the two patches done in the scope of bug #204479 are
> included in your config, especially the first one:
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/
> arch/powerpc/mm/kasan/kasan_init_32.
> c?id=663c0c9496a69f80011205ba3194049bcafd681d
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/
> arch/powerpc/mm/kasan/kasan_init_32.
> c?id=45ff3c55958542c3b76075d59741297b8cb31cbb
Ahh, now we are getting closer...

No, I did the testing with a vanilla 5.3-rc8 kernel assuming that

powerpc/kasan: Fix shadow area set up for modules
powerpc/kasan: Fix parallel loading of modules

were already in 5.3-rc8. Now I realize I was wrong.

In -rc8 the latest one (2019-07-31) is:

powerpc/kasan: fix early boot failure on PPC32

The other two patches are dated 2019-08-20 and not in 5.3-rc8 or the newly
released 5.3.

Sorry for the noise! I will close the bug for now and retest as soon as the 2
patches are in 5.4-rcx or 5.3.x. If it still seems valid that time I will
re-open.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 204819] KASAN still got problems loading some modules at boot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=204819

Michael Ellerman (michael@ellerman.id.au) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |michael@ellerman.id.au

--- Comment #26 from Michael Ellerman (michael@ellerman.id.au) ---
If you want to test things that are not quite in mainline you can always try
the "merge" branch of the powerpc tree. That is a merge of master, next and
fixes, so should have any fixes that are in the pipeline.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.