* Re: next-20190723: bpf/seccomp - systemd/journald issue?
[not found] <CA+icZUWF=B_phP8eGD3v2d9jSSK6Y-N65y-T6xewZnY91vc2_Q@mail.gmail.com>
@ 2019-07-26 15:45 ` Yonghong Song
2019-07-26 20:38 ` Sedat Dilek
2019-08-01 7:39 ` Sedat Dilek
0 siblings, 2 replies; 15+ messages in thread
From: Yonghong Song @ 2019-07-26 15:45 UTC (permalink / raw)
To: sedat.dilek, Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu
Cc: netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On 7/26/19 1:26 AM, Sedat Dilek wrote:
> Hi,
>
> I have opened a new issue in the ClangBuiltLinux issue tracker.
Glad to know clang 9 has asm goto support and now It can compile
kernel again.
>
> I am seeing a problem in the area bpf/seccomp causing
> systemd/journald/udevd services to fail.
>
> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> to connect stdout to the journal socket, ignoring: Connection refused
>
> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> BFD linker ld.bfd on Debian/buster AMD64.
> In both cases I use clang-9 (prerelease).
Looks like it is a lld bug.
I see the stack trace has __bpf_prog_run32() which is used by
kernel bpf interpreter. Could you try to enable bpf jit
sysctl net.core.bpf_jit_enable = 1
If this passed, it will prove it is interpreter related.
>
> Base for testing: next-20190723.
>
> The call-trace looks like this:
>
> [Fri Jul 26 08:08:42 2019] BUG: unable to handle page fault for
> address: ffffffff85403370
> [Fri Jul 26 08:08:42 2019] #PF: supervisor read access in kernel mode
> [Fri Jul 26 08:08:42 2019] #PF: error_code(0x0000) - not-present page
> [Fri Jul 26 08:08:42 2019] PGD 7620e067 P4D 7620e067 PUD 7620f063 PMD
> 44fe85063 PTE 800fffff8a3fc062
> [Fri Jul 26 08:08:42 2019] Oops: 0000 [#1] SMP PTI
> [Fri Jul 26 08:08:42 2019] CPU: 2 PID: 417 Comm: (journald) Not
> tainted 5.3.0-rc1-5-amd64-cbl-asmgoto #5~buster+dileks1
> [Fri Jul 26 08:08:42 2019] Hardware name: LENOVO
> 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
> [Fri Jul 26 08:08:42 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
> [Fri Jul 26 08:08:42 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
> 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
> 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 40 85 48 83 f8 3b 7f 62 48 83
> f8 1e 0f 8f c8 00
> [Fri Jul 26 08:08:42 2019] RSP: 0018:ffff992ec028fcb8 EFLAGS: 00010246
> [Fri Jul 26 08:08:42 2019] RAX: ffff992ec028fd60 RBX: ffff992ec00e9038
> RCX: 0000000000000002
> [Fri Jul 26 08:08:42 2019] RDX: ffff992ec028fd40 RSI: 00000000000000ac
> RDI: ffff992ec028fce0
> [Fri Jul 26 08:08:42 2019] RBP: ffff992ec028fcd0 R08: 0000000000000000
> R09: ffff992ec028ff58
> [Fri Jul 26 08:08:42 2019] R10: 0000000000000000 R11: ffffffff849b8210
> R12: 000000007fff0000
> [Fri Jul 26 08:08:42 2019] R13: ffff992ec028feb8 R14: 0000000000000000
> R15: ffff992ec028fce0
> [Fri Jul 26 08:08:42 2019] FS: 00007f5d20f1d940(0000)
> GS:ffff8ba3d2500000(0000) knlGS:0000000000000000
> [Fri Jul 26 08:08:42 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370 CR3: 0000000445b3e001
> CR4: 00000000003606e0
> [Fri Jul 26 08:08:42 2019] Call Trace:
> [Fri Jul 26 08:08:42 2019] __bpf_prog_run32+0x44/0x70
> [Fri Jul 26 08:08:42 2019] ? flush_tlb_func_common+0xd8/0x230
> [Fri Jul 26 08:08:42 2019] ? mem_cgroup_commit_charge+0x8c/0x120
> [Fri Jul 26 08:08:42 2019] ? wp_page_copy+0x464/0x7a0
> [Fri Jul 26 08:08:42 2019] seccomp_run_filters+0x54/0x110
> [Fri Jul 26 08:08:42 2019] __seccomp_filter+0xf7/0x6e0
> [Fri Jul 26 08:08:42 2019] ? do_wp_page+0x32b/0x5d0
> [Fri Jul 26 08:08:42 2019] ? handle_mm_fault+0x90d/0xbf0
> [Fri Jul 26 08:08:42 2019] syscall_trace_enter+0x182/0x290
> [Fri Jul 26 08:08:42 2019] do_syscall_64+0x30/0x90
> [Fri Jul 26 08:08:42 2019] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [Fri Jul 26 08:08:42 2019] RIP: 0033:0x7f5d220d7f59
> [Fri Jul 26 08:08:42 2019] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00
> 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8
> 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 07 6f 0c 00
> f7 d8 64 89 01 48
> [Fri Jul 26 08:08:42 2019] RSP: 002b:00007ffd11332b48 EFLAGS: 00000246
> ORIG_RAX: 000000000000013d
> [Fri Jul 26 08:08:42 2019] RAX: ffffffffffffffda RBX: 000055bf8ab34010
> RCX: 00007f5d220d7f59
> [Fri Jul 26 08:08:42 2019] RDX: 000055bf8ab34010 RSI: 0000000000000000
> RDI: 0000000000000001
> [Fri Jul 26 08:08:42 2019] RBP: 000055bf8ab97fb0 R08: 000055bf8abbe180
> R09: 00000000c000003e
> [Fri Jul 26 08:08:42 2019] R10: 000055bf8abbe1e0 R11: 0000000000000246
> R12: 00007ffd11332ba0
> [Fri Jul 26 08:08:42 2019] R13: 00007ffd11332b98 R14: 00007f5d21f087f8
> R15: 000000000000002c
> [Fri Jul 26 08:08:42 2019] Modules linked in: i2c_dev parport_pc
> sunrpc ppdev lp parport efivarfs ip_tables x_tables autofs4 ext4
> crc32c_generic mbcache crc16 jbd2 btrfs zstd_decompress zstd_compress
> algif_skcipher af_alg sd_mod dm_crypt dm_mod raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> raid6_pq libcrc32c raid1 uas raid0 usb_storage multipath linear
> scsi_mod md_mod hid_cherry hid_generic usbhid hid crct10dif_pclmul
> crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
> i915 glue_helper crypto_simd nvme i2c_algo_bit cryptd psmouse xhci_pci
> drm_kms_helper e1000e i2c_i801 xhci_hcd intel_lpss_pci nvme_core
> intel_lpss drm usbcore thermal wmi video button
> [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370
> [Fri Jul 26 08:08:42 2019] ---[ end trace 867b35c7d6c6705a ]---
> [Fri Jul 26 08:08:42 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
> [Fri Jul 26 08:08:42 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
> 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
> 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 40 85 48 83 f8 3b 7f 62 48 83
> f8 1e 0f 8f c8 00
> [Fri Jul 26 08:08:42 2019] RSP: 0018:ffff992ec028fcb8 EFLAGS: 00010246
> [Fri Jul 26 08:08:42 2019] RAX: ffff992ec028fd60 RBX: ffff992ec00e9038
> RCX: 0000000000000002
> [Fri Jul 26 08:08:42 2019] RDX: ffff992ec028fd40 RSI: 00000000000000ac
> RDI: ffff992ec028fce0
> [Fri Jul 26 08:08:42 2019] RBP: ffff992ec028fcd0 R08: 0000000000000000
> R09: ffff992ec028ff58
> [Fri Jul 26 08:08:42 2019] R10: 0000000000000000 R11: ffffffff849b8210
> R12: 000000007fff0000
> [Fri Jul 26 08:08:42 2019] R13: ffff992ec028feb8 R14: 0000000000000000
> R15: ffff992ec028fce0
> [Fri Jul 26 08:08:42 2019] FS: 00007f5d20f1d940(0000)
> GS:ffff8ba3d2500000(0000) knlGS:0000000000000000
> [Fri Jul 26 08:08:42 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370 CR3: 0000000445b3e001
> CR4: 00000000003606e0
>
> More details in [1] and what I tried (for example CONFIG_SECCOMP=n)
>
> I have no clue about BPF or SECCOMP.
>
> Can you comment on this?
>
> If this touches BPF: Can you give me some hints and instructions in debugging?
>
> My kernel-config and dmesg-log are attached.
>
> Thanks.
>
> Regards,
> - Sedat -
>
> [1] https://github.com/ClangBuiltLinux/linux/issues/619
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 15:45 ` next-20190723: bpf/seccomp - systemd/journald issue? Yonghong Song
@ 2019-07-26 20:38 ` Sedat Dilek
2019-07-26 21:02 ` Sedat Dilek
2019-07-26 21:05 ` Yonghong Song
2019-08-01 7:39 ` Sedat Dilek
1 sibling, 2 replies; 15+ messages in thread
From: Sedat Dilek @ 2019-07-26 20:38 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
Hi Yonghong Song,
On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > Hi,
> >
> > I have opened a new issue in the ClangBuiltLinux issue tracker.
>
> Glad to know clang 9 has asm goto support and now It can compile
> kernel again.
>
Yupp.
> >
> > I am seeing a problem in the area bpf/seccomp causing
> > systemd/journald/udevd services to fail.
> >
> > [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > to connect stdout to the journal socket, ignoring: Connection refused
> >
> > This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > BFD linker ld.bfd on Debian/buster AMD64.
> > In both cases I use clang-9 (prerelease).
>
> Looks like it is a lld bug.
>
> I see the stack trace has __bpf_prog_run32() which is used by
> kernel bpf interpreter. Could you try to enable bpf jit
> sysctl net.core.bpf_jit_enable = 1
> If this passed, it will prove it is interpreter related.
>
After...
sysctl -w net.core.bpf_jit_enable=1
I can start all failed systemd services.
systemd-journald.service
systemd-udevd.service
haveged.service
This is in maintenance mode.
What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
Regards,
- Sedat -
> >
> > Base for testing: next-20190723.
> >
> > The call-trace looks like this:
> >
> > [Fri Jul 26 08:08:42 2019] BUG: unable to handle page fault for
> > address: ffffffff85403370
> > [Fri Jul 26 08:08:42 2019] #PF: supervisor read access in kernel mode
> > [Fri Jul 26 08:08:42 2019] #PF: error_code(0x0000) - not-present page
> > [Fri Jul 26 08:08:42 2019] PGD 7620e067 P4D 7620e067 PUD 7620f063 PMD
> > 44fe85063 PTE 800fffff8a3fc062
> > [Fri Jul 26 08:08:42 2019] Oops: 0000 [#1] SMP PTI
> > [Fri Jul 26 08:08:42 2019] CPU: 2 PID: 417 Comm: (journald) Not
> > tainted 5.3.0-rc1-5-amd64-cbl-asmgoto #5~buster+dileks1
> > [Fri Jul 26 08:08:42 2019] Hardware name: LENOVO
> > 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
> > [Fri Jul 26 08:08:42 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
> > [Fri Jul 26 08:08:42 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
> > 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
> > 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 40 85 48 83 f8 3b 7f 62 48 83
> > f8 1e 0f 8f c8 00
> > [Fri Jul 26 08:08:42 2019] RSP: 0018:ffff992ec028fcb8 EFLAGS: 00010246
> > [Fri Jul 26 08:08:42 2019] RAX: ffff992ec028fd60 RBX: ffff992ec00e9038
> > RCX: 0000000000000002
> > [Fri Jul 26 08:08:42 2019] RDX: ffff992ec028fd40 RSI: 00000000000000ac
> > RDI: ffff992ec028fce0
> > [Fri Jul 26 08:08:42 2019] RBP: ffff992ec028fcd0 R08: 0000000000000000
> > R09: ffff992ec028ff58
> > [Fri Jul 26 08:08:42 2019] R10: 0000000000000000 R11: ffffffff849b8210
> > R12: 000000007fff0000
> > [Fri Jul 26 08:08:42 2019] R13: ffff992ec028feb8 R14: 0000000000000000
> > R15: ffff992ec028fce0
> > [Fri Jul 26 08:08:42 2019] FS: 00007f5d20f1d940(0000)
> > GS:ffff8ba3d2500000(0000) knlGS:0000000000000000
> > [Fri Jul 26 08:08:42 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370 CR3: 0000000445b3e001
> > CR4: 00000000003606e0
> > [Fri Jul 26 08:08:42 2019] Call Trace:
> > [Fri Jul 26 08:08:42 2019] __bpf_prog_run32+0x44/0x70
> > [Fri Jul 26 08:08:42 2019] ? flush_tlb_func_common+0xd8/0x230
> > [Fri Jul 26 08:08:42 2019] ? mem_cgroup_commit_charge+0x8c/0x120
> > [Fri Jul 26 08:08:42 2019] ? wp_page_copy+0x464/0x7a0
> > [Fri Jul 26 08:08:42 2019] seccomp_run_filters+0x54/0x110
> > [Fri Jul 26 08:08:42 2019] __seccomp_filter+0xf7/0x6e0
> > [Fri Jul 26 08:08:42 2019] ? do_wp_page+0x32b/0x5d0
> > [Fri Jul 26 08:08:42 2019] ? handle_mm_fault+0x90d/0xbf0
> > [Fri Jul 26 08:08:42 2019] syscall_trace_enter+0x182/0x290
> > [Fri Jul 26 08:08:42 2019] do_syscall_64+0x30/0x90
> > [Fri Jul 26 08:08:42 2019] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [Fri Jul 26 08:08:42 2019] RIP: 0033:0x7f5d220d7f59
> > [Fri Jul 26 08:08:42 2019] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00
> > 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8
> > 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 07 6f 0c 00
> > f7 d8 64 89 01 48
> > [Fri Jul 26 08:08:42 2019] RSP: 002b:00007ffd11332b48 EFLAGS: 00000246
> > ORIG_RAX: 000000000000013d
> > [Fri Jul 26 08:08:42 2019] RAX: ffffffffffffffda RBX: 000055bf8ab34010
> > RCX: 00007f5d220d7f59
> > [Fri Jul 26 08:08:42 2019] RDX: 000055bf8ab34010 RSI: 0000000000000000
> > RDI: 0000000000000001
> > [Fri Jul 26 08:08:42 2019] RBP: 000055bf8ab97fb0 R08: 000055bf8abbe180
> > R09: 00000000c000003e
> > [Fri Jul 26 08:08:42 2019] R10: 000055bf8abbe1e0 R11: 0000000000000246
> > R12: 00007ffd11332ba0
> > [Fri Jul 26 08:08:42 2019] R13: 00007ffd11332b98 R14: 00007f5d21f087f8
> > R15: 000000000000002c
> > [Fri Jul 26 08:08:42 2019] Modules linked in: i2c_dev parport_pc
> > sunrpc ppdev lp parport efivarfs ip_tables x_tables autofs4 ext4
> > crc32c_generic mbcache crc16 jbd2 btrfs zstd_decompress zstd_compress
> > algif_skcipher af_alg sd_mod dm_crypt dm_mod raid10 raid456
> > async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> > raid6_pq libcrc32c raid1 uas raid0 usb_storage multipath linear
> > scsi_mod md_mod hid_cherry hid_generic usbhid hid crct10dif_pclmul
> > crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
> > i915 glue_helper crypto_simd nvme i2c_algo_bit cryptd psmouse xhci_pci
> > drm_kms_helper e1000e i2c_i801 xhci_hcd intel_lpss_pci nvme_core
> > intel_lpss drm usbcore thermal wmi video button
> > [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370
> > [Fri Jul 26 08:08:42 2019] ---[ end trace 867b35c7d6c6705a ]---
> > [Fri Jul 26 08:08:42 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
> > [Fri Jul 26 08:08:42 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
> > 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
> > 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 40 85 48 83 f8 3b 7f 62 48 83
> > f8 1e 0f 8f c8 00
> > [Fri Jul 26 08:08:42 2019] RSP: 0018:ffff992ec028fcb8 EFLAGS: 00010246
> > [Fri Jul 26 08:08:42 2019] RAX: ffff992ec028fd60 RBX: ffff992ec00e9038
> > RCX: 0000000000000002
> > [Fri Jul 26 08:08:42 2019] RDX: ffff992ec028fd40 RSI: 00000000000000ac
> > RDI: ffff992ec028fce0
> > [Fri Jul 26 08:08:42 2019] RBP: ffff992ec028fcd0 R08: 0000000000000000
> > R09: ffff992ec028ff58
> > [Fri Jul 26 08:08:42 2019] R10: 0000000000000000 R11: ffffffff849b8210
> > R12: 000000007fff0000
> > [Fri Jul 26 08:08:42 2019] R13: ffff992ec028feb8 R14: 0000000000000000
> > R15: ffff992ec028fce0
> > [Fri Jul 26 08:08:42 2019] FS: 00007f5d20f1d940(0000)
> > GS:ffff8ba3d2500000(0000) knlGS:0000000000000000
> > [Fri Jul 26 08:08:42 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370 CR3: 0000000445b3e001
> > CR4: 00000000003606e0
> >
> > More details in [1] and what I tried (for example CONFIG_SECCOMP=n)
> >
> > I have no clue about BPF or SECCOMP.
> >
> > Can you comment on this?
> >
> > If this touches BPF: Can you give me some hints and instructions in debugging?
> >
> > My kernel-config and dmesg-log are attached.
> >
> > Thanks.
> >
> > Regards,
> > - Sedat -
> >
> > [1] https://github.com/ClangBuiltLinux/linux/issues/619
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 20:38 ` Sedat Dilek
@ 2019-07-26 21:02 ` Sedat Dilek
2019-07-26 21:10 ` Yonghong Song
2019-07-26 21:05 ` Yonghong Song
1 sibling, 1 reply; 15+ messages in thread
From: Sedat Dilek @ 2019-07-26 21:02 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
[-- Attachment #1: Type: text/plain, Size: 11716 bytes --]
On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> Hi Yonghong Song,
>
> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> >
> >
> >
> > On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > > Hi,
> > >
> > > I have opened a new issue in the ClangBuiltLinux issue tracker.
> >
> > Glad to know clang 9 has asm goto support and now It can compile
> > kernel again.
> >
>
> Yupp.
>
> > >
> > > I am seeing a problem in the area bpf/seccomp causing
> > > systemd/journald/udevd services to fail.
> > >
> > > [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > > to connect stdout to the journal socket, ignoring: Connection refused
> > >
> > > This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > > BFD linker ld.bfd on Debian/buster AMD64.
> > > In both cases I use clang-9 (prerelease).
> >
> > Looks like it is a lld bug.
> >
> > I see the stack trace has __bpf_prog_run32() which is used by
> > kernel bpf interpreter. Could you try to enable bpf jit
> > sysctl net.core.bpf_jit_enable = 1
> > If this passed, it will prove it is interpreter related.
> >
>
> After...
>
> sysctl -w net.core.bpf_jit_enable=1
>
> I can start all failed systemd services.
>
> systemd-journald.service
> systemd-udevd.service
> haveged.service
>
> This is in maintenance mode.
>
> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
>
This is what I did:
Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
address: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
Jul 26 22:43:06 iniza kernel: Oops: 0000 [#3] SMP PTI
Jul 26 22:43:06 iniza kernel: CPU: 3 PID: 436 Comm: systemd-udevd
Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto
#7~buster+dileks1
Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
c3 48 83 c3 08 0f b6
33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 83 f8 1e 0f 8f c8 00
Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0327a88 EFLAGS: 00010246
Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0327b30 RBX:
ffffb3cec00d1038 RCX: 0000000000000000
Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0327b10 RSI:
00000000000000ac RDI: ffffb3cec0327ab0
Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0327aa0 R08:
ffff9b33c94c0a00 R09: 0000000000000000
Jul 26 22:43:06 iniza kernel: R10: ffff9b33cfe14e00 R11:
ffffffffa77b8210 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
0000000000000000 R15: ffffb3cec0327ab0
Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
GS:ffff9b33d2580000(0000) knlGS:0000000000000000
Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
000000044f3ea006 CR4: 00000000003606e0
Jul 26 22:43:06 iniza kernel: Call Trace:
Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70
Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60
Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220
Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100
Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0
Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0
Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330
Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110
Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490
Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0
Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0
Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0
Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0
Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0
Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90
Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
89 d4 55 48 89 f5 53
Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
00000246 ORIG_RAX: 000000000000002e
Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
0000561e28ac9390 RCX: 00007f7ac3519914
Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
00007ffcfb66a4a0 RDI: 000000000000000d
Jul 26 22:43:06 iniza kernel: RBP: 0000561e28acd210 R08:
0000561e28990140 R09: 0000000000000002
Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
0000000000000246 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
000000000000005e R15: 00007ffcfb66a490
Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
button
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0391 ]---
Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
83 f8 1e 0f 8f c8 00
Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0253cb8 EFLAGS: 00010246
Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0253d60 RBX:
ffffb3cec00e9038 RCX: 0000000000000002
Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0253d40 RSI:
00000000000000ac RDI: ffffb3cec0253ce0
Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0253cd0 R08:
0000000000000000 R09: ffffb3cec0253f58
Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
ffffffffa77b8210 R12: 000000007fff0000
Jul 26 22:43:06 iniza kernel: R13: ffffb3cec0253eb8 R14:
0000000000000000 R15: ffffb3cec0253ce0
Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
GS:ffff9b33d2580000(0000) knlGS:0000000000000000
Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
000000044f3ea006 CR4: 00000000003606e0
Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
address: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
Jul 26 22:43:06 iniza kernel: Oops: 0000 [#4] SMP PTI
Jul 26 22:43:06 iniza kernel: CPU: 0 PID: 437 Comm: systemd-udevd
Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto
#7~buster+dileks1
Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
83 f8 1e 0f 8f c8 00
Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec032fa88 EFLAGS: 00010246
Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec032fb30 RBX:
ffffb3cec00d1038 RCX: 0000000000000000
Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec032fb10 RSI:
00000000000000ac RDI: ffffb3cec032fab0
Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec032faa0 R08:
ffff9b33cf34b000 R09: 0000000000000000
Jul 26 22:43:06 iniza kernel: R10: ffff9b33cf3a3400 R11:
ffffffffa77b8210 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
0000000000000000 R15: ffffb3cec032fab0
Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
GS:ffff9b33d2400000(0000) knlGS:0000000000000000
Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
000000044724a001 CR4: 00000000003606f0
Jul 26 22:43:06 iniza kernel: Call Trace:
Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70
Jul 26 22:43:06 iniza kernel: ? prep_new_page+0x47/0x1a0
Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60
Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220
Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100
Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0
Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0
Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330
Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110
Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490
Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0
Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0
Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0
Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0
Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0
Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90
Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
89 d4 55 48 89 f5 53
Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
00000246 ORIG_RAX: 000000000000002e
Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
0000561e28aaa600 RCX: 00007f7ac3519914
Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
00007ffcfb66a4a0 RDI: 000000000000000e
Jul 26 22:43:06 iniza kernel: RBP: 0000561e28aaaac0 R08:
0000561e28990140 R09: 0000000000000002
Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
0000000000000246 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
000000000000005d R15: 00007ffcfb66a490
Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
button
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0392 ]---
Full `journalctl -xb` attached.
- Sedat -
[-- Attachment #2: journalctl-xb.txt.gz --]
[-- Type: application/gzip, Size: 26676 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 20:38 ` Sedat Dilek
2019-07-26 21:02 ` Sedat Dilek
@ 2019-07-26 21:05 ` Yonghong Song
1 sibling, 0 replies; 15+ messages in thread
From: Yonghong Song @ 2019-07-26 21:05 UTC (permalink / raw)
To: sedat.dilek
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On 7/26/19 1:38 PM, Sedat Dilek wrote:
> Hi Yonghong Song,
>
> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
>>
>>
>>
>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
>>> Hi,
>>>
>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
>>
>> Glad to know clang 9 has asm goto support and now It can compile
>> kernel again.
>>
>
> Yupp.
>
>>>
>>> I am seeing a problem in the area bpf/seccomp causing
>>> systemd/journald/udevd services to fail.
>>>
>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
>>> to connect stdout to the journal socket, ignoring: Connection refused
>>>
>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
>>> BFD linker ld.bfd on Debian/buster AMD64.
>>> In both cases I use clang-9 (prerelease).
>>
>> Looks like it is a lld bug.
>>
>> I see the stack trace has __bpf_prog_run32() which is used by
>> kernel bpf interpreter. Could you try to enable bpf jit
>> sysctl net.core.bpf_jit_enable = 1
>> If this passed, it will prove it is interpreter related.
>>
>
> After...
>
> sysctl -w net.core.bpf_jit_enable=1
>
> I can start all failed systemd services.
>
> systemd-journald.service
> systemd-udevd.service
> haveged.service
>
> This is in maintenance mode.
>
> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
I do think you should set net.core.bpf_jit_enable by default. This is
more tested in production and you get better performance as well.
>
> Regards,
> - Sedat -
>
>>>
>>> Base for testing: next-20190723.
>>>
>>> The call-trace looks like this:
>>>
>>> [Fri Jul 26 08:08:42 2019] BUG: unable to handle page fault for
>>> address: ffffffff85403370
>>> [Fri Jul 26 08:08:42 2019] #PF: supervisor read access in kernel mode
>>> [Fri Jul 26 08:08:42 2019] #PF: error_code(0x0000) - not-present page
>>> [Fri Jul 26 08:08:42 2019] PGD 7620e067 P4D 7620e067 PUD 7620f063 PMD
>>> 44fe85063 PTE 800fffff8a3fc062
>>> [Fri Jul 26 08:08:42 2019] Oops: 0000 [#1] SMP PTI
>>> [Fri Jul 26 08:08:42 2019] CPU: 2 PID: 417 Comm: (journald) Not
>>> tainted 5.3.0-rc1-5-amd64-cbl-asmgoto #5~buster+dileks1
>>> [Fri Jul 26 08:08:42 2019] Hardware name: LENOVO
>>> 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
>>> [Fri Jul 26 08:08:42 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
>>> [Fri Jul 26 08:08:42 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
>>> 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
>>> 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 40 85 48 83 f8 3b 7f 62 48 83
>>> f8 1e 0f 8f c8 00
>>> [Fri Jul 26 08:08:42 2019] RSP: 0018:ffff992ec028fcb8 EFLAGS: 00010246
>>> [Fri Jul 26 08:08:42 2019] RAX: ffff992ec028fd60 RBX: ffff992ec00e9038
>>> RCX: 0000000000000002
>>> [Fri Jul 26 08:08:42 2019] RDX: ffff992ec028fd40 RSI: 00000000000000ac
>>> RDI: ffff992ec028fce0
>>> [Fri Jul 26 08:08:42 2019] RBP: ffff992ec028fcd0 R08: 0000000000000000
>>> R09: ffff992ec028ff58
>>> [Fri Jul 26 08:08:42 2019] R10: 0000000000000000 R11: ffffffff849b8210
>>> R12: 000000007fff0000
>>> [Fri Jul 26 08:08:42 2019] R13: ffff992ec028feb8 R14: 0000000000000000
>>> R15: ffff992ec028fce0
>>> [Fri Jul 26 08:08:42 2019] FS: 00007f5d20f1d940(0000)
>>> GS:ffff8ba3d2500000(0000) knlGS:0000000000000000
>>> [Fri Jul 26 08:08:42 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370 CR3: 0000000445b3e001
>>> CR4: 00000000003606e0
>>> [Fri Jul 26 08:08:42 2019] Call Trace:
>>> [Fri Jul 26 08:08:42 2019] __bpf_prog_run32+0x44/0x70
>>> [Fri Jul 26 08:08:42 2019] ? flush_tlb_func_common+0xd8/0x230
>>> [Fri Jul 26 08:08:42 2019] ? mem_cgroup_commit_charge+0x8c/0x120
>>> [Fri Jul 26 08:08:42 2019] ? wp_page_copy+0x464/0x7a0
>>> [Fri Jul 26 08:08:42 2019] seccomp_run_filters+0x54/0x110
>>> [Fri Jul 26 08:08:42 2019] __seccomp_filter+0xf7/0x6e0
>>> [Fri Jul 26 08:08:42 2019] ? do_wp_page+0x32b/0x5d0
>>> [Fri Jul 26 08:08:42 2019] ? handle_mm_fault+0x90d/0xbf0
>>> [Fri Jul 26 08:08:42 2019] syscall_trace_enter+0x182/0x290
>>> [Fri Jul 26 08:08:42 2019] do_syscall_64+0x30/0x90
>>> [Fri Jul 26 08:08:42 2019] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [Fri Jul 26 08:08:42 2019] RIP: 0033:0x7f5d220d7f59
>>> [Fri Jul 26 08:08:42 2019] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00
>>> 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8
>>> 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 07 6f 0c 00
>>> f7 d8 64 89 01 48
>>> [Fri Jul 26 08:08:42 2019] RSP: 002b:00007ffd11332b48 EFLAGS: 00000246
>>> ORIG_RAX: 000000000000013d
>>> [Fri Jul 26 08:08:42 2019] RAX: ffffffffffffffda RBX: 000055bf8ab34010
>>> RCX: 00007f5d220d7f59
>>> [Fri Jul 26 08:08:42 2019] RDX: 000055bf8ab34010 RSI: 0000000000000000
>>> RDI: 0000000000000001
>>> [Fri Jul 26 08:08:42 2019] RBP: 000055bf8ab97fb0 R08: 000055bf8abbe180
>>> R09: 00000000c000003e
>>> [Fri Jul 26 08:08:42 2019] R10: 000055bf8abbe1e0 R11: 0000000000000246
>>> R12: 00007ffd11332ba0
>>> [Fri Jul 26 08:08:42 2019] R13: 00007ffd11332b98 R14: 00007f5d21f087f8
>>> R15: 000000000000002c
>>> [Fri Jul 26 08:08:42 2019] Modules linked in: i2c_dev parport_pc
>>> sunrpc ppdev lp parport efivarfs ip_tables x_tables autofs4 ext4
>>> crc32c_generic mbcache crc16 jbd2 btrfs zstd_decompress zstd_compress
>>> algif_skcipher af_alg sd_mod dm_crypt dm_mod raid10 raid456
>>> async_raid6_recov async_memcpy async_pq async_xor async_tx xor
>>> raid6_pq libcrc32c raid1 uas raid0 usb_storage multipath linear
>>> scsi_mod md_mod hid_cherry hid_generic usbhid hid crct10dif_pclmul
>>> crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
>>> i915 glue_helper crypto_simd nvme i2c_algo_bit cryptd psmouse xhci_pci
>>> drm_kms_helper e1000e i2c_i801 xhci_hcd intel_lpss_pci nvme_core
>>> intel_lpss drm usbcore thermal wmi video button
>>> [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370
>>> [Fri Jul 26 08:08:42 2019] ---[ end trace 867b35c7d6c6705a ]---
>>> [Fri Jul 26 08:08:42 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
>>> [Fri Jul 26 08:08:42 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
>>> 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
>>> 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 40 85 48 83 f8 3b 7f 62 48 83
>>> f8 1e 0f 8f c8 00
>>> [Fri Jul 26 08:08:42 2019] RSP: 0018:ffff992ec028fcb8 EFLAGS: 00010246
>>> [Fri Jul 26 08:08:42 2019] RAX: ffff992ec028fd60 RBX: ffff992ec00e9038
>>> RCX: 0000000000000002
>>> [Fri Jul 26 08:08:42 2019] RDX: ffff992ec028fd40 RSI: 00000000000000ac
>>> RDI: ffff992ec028fce0
>>> [Fri Jul 26 08:08:42 2019] RBP: ffff992ec028fcd0 R08: 0000000000000000
>>> R09: ffff992ec028ff58
>>> [Fri Jul 26 08:08:42 2019] R10: 0000000000000000 R11: ffffffff849b8210
>>> R12: 000000007fff0000
>>> [Fri Jul 26 08:08:42 2019] R13: ffff992ec028feb8 R14: 0000000000000000
>>> R15: ffff992ec028fce0
>>> [Fri Jul 26 08:08:42 2019] FS: 00007f5d20f1d940(0000)
>>> GS:ffff8ba3d2500000(0000) knlGS:0000000000000000
>>> [Fri Jul 26 08:08:42 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [Fri Jul 26 08:08:42 2019] CR2: ffffffff85403370 CR3: 0000000445b3e001
>>> CR4: 00000000003606e0
>>>
>>> More details in [1] and what I tried (for example CONFIG_SECCOMP=n)
>>>
>>> I have no clue about BPF or SECCOMP.
>>>
>>> Can you comment on this?
>>>
>>> If this touches BPF: Can you give me some hints and instructions in debugging?
>>>
>>> My kernel-config and dmesg-log are attached.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> - Sedat -
>>>
>>> [1] https://github.com/ClangBuiltLinux/linux/issues/619
>>>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 21:02 ` Sedat Dilek
@ 2019-07-26 21:10 ` Yonghong Song
2019-07-26 21:19 ` Sedat Dilek
0 siblings, 1 reply; 15+ messages in thread
From: Yonghong Song @ 2019-07-26 21:10 UTC (permalink / raw)
To: sedat.dilek
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On 7/26/19 2:02 PM, Sedat Dilek wrote:
> On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> Hi Yonghong Song,
>>
>> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
>>>
>>>
>>>
>>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
>>>> Hi,
>>>>
>>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
>>>
>>> Glad to know clang 9 has asm goto support and now It can compile
>>> kernel again.
>>>
>>
>> Yupp.
>>
>>>>
>>>> I am seeing a problem in the area bpf/seccomp causing
>>>> systemd/journald/udevd services to fail.
>>>>
>>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
>>>> to connect stdout to the journal socket, ignoring: Connection refused
>>>>
>>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
>>>> BFD linker ld.bfd on Debian/buster AMD64.
>>>> In both cases I use clang-9 (prerelease).
>>>
>>> Looks like it is a lld bug.
>>>
>>> I see the stack trace has __bpf_prog_run32() which is used by
>>> kernel bpf interpreter. Could you try to enable bpf jit
>>> sysctl net.core.bpf_jit_enable = 1
>>> If this passed, it will prove it is interpreter related.
>>>
>>
>> After...
>>
>> sysctl -w net.core.bpf_jit_enable=1
>>
>> I can start all failed systemd services.
>>
>> systemd-journald.service
>> systemd-udevd.service
>> haveged.service
>>
>> This is in maintenance mode.
>>
>> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
>>
>
> This is what I did:
I probably won't have cycles to debug this potential lld issue.
Maybe you already did, I suggest you put enough reproducible
details in the bug you filed against lld so they can take a look.
>
> Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
> address: ffffffffa8203370
> Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
> Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
> Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
> 2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
> Jul 26 22:43:06 iniza kernel: Oops: 0000 [#3] SMP PTI
> Jul 26 22:43:06 iniza kernel: CPU: 3 PID: 436 Comm: systemd-udevd
> Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto
> #7~buster+dileks1
> Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
> 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
> Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
> Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
> 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
> c3 48 83 c3 08 0f b6
> 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 83 f8 1e 0f 8f c8 00
> Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0327a88 EFLAGS: 00010246
> Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0327b30 RBX:
> ffffb3cec00d1038 RCX: 0000000000000000
> Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0327b10 RSI:
> 00000000000000ac RDI: ffffb3cec0327ab0
> Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0327aa0 R08:
> ffff9b33c94c0a00 R09: 0000000000000000
> Jul 26 22:43:06 iniza kernel: R10: ffff9b33cfe14e00 R11:
> ffffffffa77b8210 R12: 0000000000000000
> Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
> 0000000000000000 R15: ffffb3cec0327ab0
> Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
> GS:ffff9b33d2580000(0000) knlGS:0000000000000000
> Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
> 000000044f3ea006 CR4: 00000000003606e0
> Jul 26 22:43:06 iniza kernel: Call Trace:
> Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70
> Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60
> Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220
> Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100
> Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0
> Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0
> Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330
> Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110
> Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490
> Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0
> Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0
> Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0
> Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0
> Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0
> Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90
> Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
> Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
> ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
> 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
> 89 d4 55 48 89 f5 53
> Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
> 00000246 ORIG_RAX: 000000000000002e
> Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
> 0000561e28ac9390 RCX: 00007f7ac3519914
> Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
> 00007ffcfb66a4a0 RDI: 000000000000000d
> Jul 26 22:43:06 iniza kernel: RBP: 0000561e28acd210 R08:
> 0000561e28990140 R09: 0000000000000002
> Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
> 0000000000000246 R12: 0000000000000000
> Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
> 000000000000005e R15: 00007ffcfb66a490
> Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
> nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
> efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
> jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
> uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
> raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
> aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
> i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
> xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
> button
> Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
> Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0391 ]---
> Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
> Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
> 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
> c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
> 83 f8 1e 0f 8f c8 00
> Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0253cb8 EFLAGS: 00010246
> Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0253d60 RBX:
> ffffb3cec00e9038 RCX: 0000000000000002
> Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0253d40 RSI:
> 00000000000000ac RDI: ffffb3cec0253ce0
> Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0253cd0 R08:
> 0000000000000000 R09: ffffb3cec0253f58
> Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
> ffffffffa77b8210 R12: 000000007fff0000
> Jul 26 22:43:06 iniza kernel: R13: ffffb3cec0253eb8 R14:
> 0000000000000000 R15: ffffb3cec0253ce0
> Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
> GS:ffff9b33d2580000(0000) knlGS:0000000000000000
> Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
> 000000044f3ea006 CR4: 00000000003606e0
> Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
> address: ffffffffa8203370
> Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
> Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
> Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
> 2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
> Jul 26 22:43:06 iniza kernel: Oops: 0000 [#4] SMP PTI
> Jul 26 22:43:06 iniza kernel: CPU: 0 PID: 437 Comm: systemd-udevd
> Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto
> #7~buster+dileks1
> Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
> 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
> Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
> Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
> 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
> c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
> 83 f8 1e 0f 8f c8 00
> Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec032fa88 EFLAGS: 00010246
> Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec032fb30 RBX:
> ffffb3cec00d1038 RCX: 0000000000000000
> Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec032fb10 RSI:
> 00000000000000ac RDI: ffffb3cec032fab0
> Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec032faa0 R08:
> ffff9b33cf34b000 R09: 0000000000000000
> Jul 26 22:43:06 iniza kernel: R10: ffff9b33cf3a3400 R11:
> ffffffffa77b8210 R12: 0000000000000000
> Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
> 0000000000000000 R15: ffffb3cec032fab0
> Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
> GS:ffff9b33d2400000(0000) knlGS:0000000000000000
> Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
> 000000044724a001 CR4: 00000000003606f0
> Jul 26 22:43:06 iniza kernel: Call Trace:
> Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70
> Jul 26 22:43:06 iniza kernel: ? prep_new_page+0x47/0x1a0
> Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60
> Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220
> Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100
> Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0
> Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0
> Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330
> Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110
> Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490
> Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0
> Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0
> Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0
> Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0
> Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0
> Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90
> Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
> Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
> ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
> 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
> 89 d4 55 48 89 f5 53
> Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
> 00000246 ORIG_RAX: 000000000000002e
> Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
> 0000561e28aaa600 RCX: 00007f7ac3519914
> Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
> 00007ffcfb66a4a0 RDI: 000000000000000e
> Jul 26 22:43:06 iniza kernel: RBP: 0000561e28aaaac0 R08:
> 0000561e28990140 R09: 0000000000000002
> Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
> 0000000000000246 R12: 0000000000000000
> Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
> 000000000000005d R15: 00007ffcfb66a490
> Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
> nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
> efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
> jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
> uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
> raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
> aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
> i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
> xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
> button
> Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
> Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0392 ]---
>
> Full `journalctl -xb` attached.
>
> - Sedat -
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 21:10 ` Yonghong Song
@ 2019-07-26 21:19 ` Sedat Dilek
2019-07-27 2:24 ` Alexei Starovoitov
0 siblings, 1 reply; 15+ messages in thread
From: Sedat Dilek @ 2019-07-26 21:19 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 7/26/19 2:02 PM, Sedat Dilek wrote:
> > On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>
> >> Hi Yonghong Song,
> >>
> >> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> >>>
> >>>
> >>>
> >>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> >>>> Hi,
> >>>>
> >>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> >>>
> >>> Glad to know clang 9 has asm goto support and now It can compile
> >>> kernel again.
> >>>
> >>
> >> Yupp.
> >>
> >>>>
> >>>> I am seeing a problem in the area bpf/seccomp causing
> >>>> systemd/journald/udevd services to fail.
> >>>>
> >>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> >>>> to connect stdout to the journal socket, ignoring: Connection refused
> >>>>
> >>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> >>>> BFD linker ld.bfd on Debian/buster AMD64.
> >>>> In both cases I use clang-9 (prerelease).
> >>>
> >>> Looks like it is a lld bug.
> >>>
> >>> I see the stack trace has __bpf_prog_run32() which is used by
> >>> kernel bpf interpreter. Could you try to enable bpf jit
> >>> sysctl net.core.bpf_jit_enable = 1
> >>> If this passed, it will prove it is interpreter related.
> >>>
> >>
> >> After...
> >>
> >> sysctl -w net.core.bpf_jit_enable=1
> >>
> >> I can start all failed systemd services.
> >>
> >> systemd-journald.service
> >> systemd-udevd.service
> >> haveged.service
> >>
> >> This is in maintenance mode.
> >>
> >> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> >>
> >
> > This is what I did:
>
> I probably won't have cycles to debug this potential lld issue.
> Maybe you already did, I suggest you put enough reproducible
> details in the bug you filed against lld so they can take a look.
>
I understand and will put the journalctl-log into the CBL issue
tracker and update informations.
Thanks for your help understanding the BPF correlations.
Is setting 'net.core.bpf_jit_enable = 2' helpful here?
Values :
0 - disable the JIT (default value)
1 - enable the JIT
2 - enable the JIT and ask the compiler to emit traces on kernel log.
Which files should LLD folks look at?
cd linux
find ./ -name '*bpf*.o' | grep jit
./arch/x86/net/bpf_jit_comp.o
Compare the objdumps?
I have archived the full build-dirs of clang9+ld.bfd and clang9+ld.lld-9.
Thanks for your help!
- sed@ -
[1] https://sysctl-explorer.net/net/core/bpf_jit_enable/
> >
> > Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
> > address: ffffffffa8203370
> > Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
> > Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
> > Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
> > 2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
> > Jul 26 22:43:06 iniza kernel: Oops: 0000 [#3] SMP PTI
> > Jul 26 22:43:06 iniza kernel: CPU: 3 PID: 436 Comm: systemd-udevd
> > Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto
> > #7~buster+dileks1
> > Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
> > 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
> > Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
> > Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
> > 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
> > c3 48 83 c3 08 0f b6
> > 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 83 f8 1e 0f 8f c8 00
> > Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0327a88 EFLAGS: 00010246
> > Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0327b30 RBX:
> > ffffb3cec00d1038 RCX: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0327b10 RSI:
> > 00000000000000ac RDI: ffffb3cec0327ab0
> > Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0327aa0 R08:
> > ffff9b33c94c0a00 R09: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: R10: ffff9b33cfe14e00 R11:
> > ffffffffa77b8210 R12: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
> > 0000000000000000 R15: ffffb3cec0327ab0
> > Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
> > GS:ffff9b33d2580000(0000) knlGS:0000000000000000
> > Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
> > 000000044f3ea006 CR4: 00000000003606e0
> > Jul 26 22:43:06 iniza kernel: Call Trace:
> > Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70
> > Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60
> > Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220
> > Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100
> > Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0
> > Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0
> > Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330
> > Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110
> > Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490
> > Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0
> > Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0
> > Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0
> > Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0
> > Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0
> > Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90
> > Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
> > Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
> > ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
> > 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
> > 89 d4 55 48 89 f5 53
> > Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
> > 00000246 ORIG_RAX: 000000000000002e
> > Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
> > 0000561e28ac9390 RCX: 00007f7ac3519914
> > Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
> > 00007ffcfb66a4a0 RDI: 000000000000000d
> > Jul 26 22:43:06 iniza kernel: RBP: 0000561e28acd210 R08:
> > 0000561e28990140 R09: 0000000000000002
> > Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
> > 0000000000000246 R12: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
> > 000000000000005e R15: 00007ffcfb66a490
> > Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
> > nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
> > efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
> > jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
> > uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
> > raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> > raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
> > crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
> > aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
> > i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
> > xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
> > button
> > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
> > Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0391 ]---
> > Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
> > Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
> > 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
> > c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
> > 83 f8 1e 0f 8f c8 00
> > Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0253cb8 EFLAGS: 00010246
> > Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0253d60 RBX:
> > ffffb3cec00e9038 RCX: 0000000000000002
> > Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0253d40 RSI:
> > 00000000000000ac RDI: ffffb3cec0253ce0
> > Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0253cd0 R08:
> > 0000000000000000 R09: ffffb3cec0253f58
> > Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
> > ffffffffa77b8210 R12: 000000007fff0000
> > Jul 26 22:43:06 iniza kernel: R13: ffffb3cec0253eb8 R14:
> > 0000000000000000 R15: ffffb3cec0253ce0
> > Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
> > GS:ffff9b33d2580000(0000) knlGS:0000000000000000
> > Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
> > 000000044f3ea006 CR4: 00000000003606e0
> > Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
> > address: ffffffffa8203370
> > Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
> > Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
> > Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
> > 2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
> > Jul 26 22:43:06 iniza kernel: Oops: 0000 [#4] SMP PTI
> > Jul 26 22:43:06 iniza kernel: CPU: 0 PID: 437 Comm: systemd-udevd
> > Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto
> > #7~buster+dileks1
> > Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
> > 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
> > Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
> > Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
> > 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
> > c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
> > 83 f8 1e 0f 8f c8 00
> > Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec032fa88 EFLAGS: 00010246
> > Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec032fb30 RBX:
> > ffffb3cec00d1038 RCX: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec032fb10 RSI:
> > 00000000000000ac RDI: ffffb3cec032fab0
> > Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec032faa0 R08:
> > ffff9b33cf34b000 R09: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: R10: ffff9b33cf3a3400 R11:
> > ffffffffa77b8210 R12: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
> > 0000000000000000 R15: ffffb3cec032fab0
> > Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000)
> > GS:ffff9b33d2400000(0000) knlGS:0000000000000000
> > Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
> > 000000044724a001 CR4: 00000000003606f0
> > Jul 26 22:43:06 iniza kernel: Call Trace:
> > Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70
> > Jul 26 22:43:06 iniza kernel: ? prep_new_page+0x47/0x1a0
> > Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60
> > Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220
> > Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100
> > Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0
> > Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0
> > Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330
> > Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110
> > Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490
> > Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0
> > Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0
> > Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0
> > Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0
> > Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0
> > Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90
> > Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
> > Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
> > ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
> > 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
> > 89 d4 55 48 89 f5 53
> > Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
> > 00000246 ORIG_RAX: 000000000000002e
> > Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
> > 0000561e28aaa600 RCX: 00007f7ac3519914
> > Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
> > 00007ffcfb66a4a0 RDI: 000000000000000e
> > Jul 26 22:43:06 iniza kernel: RBP: 0000561e28aaaac0 R08:
> > 0000561e28990140 R09: 0000000000000002
> > Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
> > 0000000000000246 R12: 0000000000000000
> > Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
> > 000000000000005d R15: 00007ffcfb66a490
> > Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
> > nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
> > efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
> > jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
> > uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
> > raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> > raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
> > crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
> > aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
> > i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
> > xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
> > button
> > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
> > Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0392 ]---
> >
> > Full `journalctl -xb` attached.
> >
> > - Sedat -
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 21:19 ` Sedat Dilek
@ 2019-07-27 2:24 ` Alexei Starovoitov
2019-07-27 7:36 ` Sedat Dilek
0 siblings, 1 reply; 15+ messages in thread
From: Alexei Starovoitov @ 2019-07-27 2:24 UTC (permalink / raw)
To: sedat.dilek
Cc: Yonghong Song, Alexei Starovoitov, Daniel Borkmann, Martin Lau,
Song Liu, netdev, bpf, Clang-Built-Linux ML, Kees Cook,
Nick Desaulniers, Nathan Chancellor
On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
> >
> >
> >
> > On 7/26/19 2:02 PM, Sedat Dilek wrote:
> > > On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >>
> > >> Hi Yonghong Song,
> > >>
> > >> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> > >>>
> > >>>
> > >>>
> > >>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> > >>>
> > >>> Glad to know clang 9 has asm goto support and now It can compile
> > >>> kernel again.
> > >>>
> > >>
> > >> Yupp.
> > >>
> > >>>>
> > >>>> I am seeing a problem in the area bpf/seccomp causing
> > >>>> systemd/journald/udevd services to fail.
> > >>>>
> > >>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > >>>> to connect stdout to the journal socket, ignoring: Connection refused
> > >>>>
> > >>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > >>>> BFD linker ld.bfd on Debian/buster AMD64.
> > >>>> In both cases I use clang-9 (prerelease).
> > >>>
> > >>> Looks like it is a lld bug.
> > >>>
> > >>> I see the stack trace has __bpf_prog_run32() which is used by
> > >>> kernel bpf interpreter. Could you try to enable bpf jit
> > >>> sysctl net.core.bpf_jit_enable = 1
> > >>> If this passed, it will prove it is interpreter related.
> > >>>
> > >>
> > >> After...
> > >>
> > >> sysctl -w net.core.bpf_jit_enable=1
> > >>
> > >> I can start all failed systemd services.
> > >>
> > >> systemd-journald.service
> > >> systemd-udevd.service
> > >> haveged.service
> > >>
> > >> This is in maintenance mode.
> > >>
> > >> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> > >>
> > >
> > > This is what I did:
> >
> > I probably won't have cycles to debug this potential lld issue.
> > Maybe you already did, I suggest you put enough reproducible
> > details in the bug you filed against lld so they can take a look.
> >
>
> I understand and will put the journalctl-log into the CBL issue
> tracker and update informations.
>
> Thanks for your help understanding the BPF correlations.
>
> Is setting 'net.core.bpf_jit_enable = 2' helpful here?
jit_enable=1 is enough.
Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
It sounds like clang miscompiles interpreter.
modprobe test_bpf
should be able to point out which part of interpreter is broken.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-27 2:24 ` Alexei Starovoitov
@ 2019-07-27 7:36 ` Sedat Dilek
2019-07-27 8:16 ` Sedat Dilek
2019-07-27 17:08 ` Yonghong Song
0 siblings, 2 replies; 15+ messages in thread
From: Sedat Dilek @ 2019-07-27 7:36 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Yonghong Song, Alexei Starovoitov, Daniel Borkmann, Martin Lau,
Song Liu, netdev, bpf, Clang-Built-Linux ML, Kees Cook,
Nick Desaulniers, Nathan Chancellor
On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
> > >
> > >
> > >
> > > On 7/26/19 2:02 PM, Sedat Dilek wrote:
> > > > On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > >>
> > > >> Hi Yonghong Song,
> > > >>
> > > >> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> > > >>>
> > > >>> Glad to know clang 9 has asm goto support and now It can compile
> > > >>> kernel again.
> > > >>>
> > > >>
> > > >> Yupp.
> > > >>
> > > >>>>
> > > >>>> I am seeing a problem in the area bpf/seccomp causing
> > > >>>> systemd/journald/udevd services to fail.
> > > >>>>
> > > >>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > > >>>> to connect stdout to the journal socket, ignoring: Connection refused
> > > >>>>
> > > >>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > > >>>> BFD linker ld.bfd on Debian/buster AMD64.
> > > >>>> In both cases I use clang-9 (prerelease).
> > > >>>
> > > >>> Looks like it is a lld bug.
> > > >>>
> > > >>> I see the stack trace has __bpf_prog_run32() which is used by
> > > >>> kernel bpf interpreter. Could you try to enable bpf jit
> > > >>> sysctl net.core.bpf_jit_enable = 1
> > > >>> If this passed, it will prove it is interpreter related.
> > > >>>
> > > >>
> > > >> After...
> > > >>
> > > >> sysctl -w net.core.bpf_jit_enable=1
> > > >>
> > > >> I can start all failed systemd services.
> > > >>
> > > >> systemd-journald.service
> > > >> systemd-udevd.service
> > > >> haveged.service
> > > >>
> > > >> This is in maintenance mode.
> > > >>
> > > >> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> > > >>
> > > >
> > > > This is what I did:
> > >
> > > I probably won't have cycles to debug this potential lld issue.
> > > Maybe you already did, I suggest you put enough reproducible
> > > details in the bug you filed against lld so they can take a look.
> > >
> >
> > I understand and will put the journalctl-log into the CBL issue
> > tracker and update informations.
> >
> > Thanks for your help understanding the BPF correlations.
> >
> > Is setting 'net.core.bpf_jit_enable = 2' helpful here?
>
> jit_enable=1 is enough.
> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
>
> It sounds like clang miscompiles interpreter.
> modprobe test_bpf
> should be able to point out which part of interpreter is broken.
Maybe we need something like...
"bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
...for clang?
- Sedat -
[1] https://git.kernel.org/linus/3193c0836f203a91bef96d88c64cccf0be090d9c
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-27 7:36 ` Sedat Dilek
@ 2019-07-27 8:16 ` Sedat Dilek
2019-07-27 17:11 ` Yonghong Song
2019-07-27 17:08 ` Yonghong Song
1 sibling, 1 reply; 15+ messages in thread
From: Sedat Dilek @ 2019-07-27 8:16 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Yonghong Song, Alexei Starovoitov, Daniel Borkmann, Martin Lau,
Song Liu, netdev, bpf, Clang-Built-Linux ML, Kees Cook,
Nick Desaulniers, Nathan Chancellor
On Sat, Jul 27, 2019 at 9:36 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
> > > >
> > > >
> > > >
> > > > On 7/26/19 2:02 PM, Sedat Dilek wrote:
> > > > > On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > > >>
> > > > >> Hi Yonghong Song,
> > > > >>
> > > > >> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> > > > >>>
> > > > >>> Glad to know clang 9 has asm goto support and now It can compile
> > > > >>> kernel again.
> > > > >>>
> > > > >>
> > > > >> Yupp.
> > > > >>
> > > > >>>>
> > > > >>>> I am seeing a problem in the area bpf/seccomp causing
> > > > >>>> systemd/journald/udevd services to fail.
> > > > >>>>
> > > > >>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > > > >>>> to connect stdout to the journal socket, ignoring: Connection refused
> > > > >>>>
> > > > >>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > > > >>>> BFD linker ld.bfd on Debian/buster AMD64.
> > > > >>>> In both cases I use clang-9 (prerelease).
> > > > >>>
> > > > >>> Looks like it is a lld bug.
> > > > >>>
> > > > >>> I see the stack trace has __bpf_prog_run32() which is used by
> > > > >>> kernel bpf interpreter. Could you try to enable bpf jit
> > > > >>> sysctl net.core.bpf_jit_enable = 1
> > > > >>> If this passed, it will prove it is interpreter related.
> > > > >>>
> > > > >>
> > > > >> After...
> > > > >>
> > > > >> sysctl -w net.core.bpf_jit_enable=1
> > > > >>
> > > > >> I can start all failed systemd services.
> > > > >>
> > > > >> systemd-journald.service
> > > > >> systemd-udevd.service
> > > > >> haveged.service
> > > > >>
> > > > >> This is in maintenance mode.
> > > > >>
> > > > >> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> > > > >>
> > > > >
> > > > > This is what I did:
> > > >
> > > > I probably won't have cycles to debug this potential lld issue.
> > > > Maybe you already did, I suggest you put enough reproducible
> > > > details in the bug you filed against lld so they can take a look.
> > > >
> > >
> > > I understand and will put the journalctl-log into the CBL issue
> > > tracker and update informations.
> > >
> > > Thanks for your help understanding the BPF correlations.
> > >
> > > Is setting 'net.core.bpf_jit_enable = 2' helpful here?
> >
> > jit_enable=1 is enough.
> > Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
> >
> > It sounds like clang miscompiles interpreter.
Just to clarify:
This does not happen with clang-9 + ld.bfd (GNU/ld linker).
> > modprobe test_bpf
> > should be able to point out which part of interpreter is broken.
>
> Maybe we need something like...
>
> "bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
>
> ...for clang?
>
Not sure if something like GCC's...
-fgcse
Perform a global common subexpression elimination pass. This pass also
performs global constant and copy propagation.
Note: When compiling a program using computed gotos, a GCC extension,
you may get better run-time performance if you disable the global
common subexpression elimination pass by adding -fno-gcse to the
command line.
Enabled at levels -O2, -O3, -Os.
...is available for clang.
I tried with hopping to turn off "global common subexpression elimination":
diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
index 383c87300b0d..92f934a1e9ff 100644
--- a/arch/x86/net/Makefile
+++ b/arch/x86/net/Makefile
@@ -3,6 +3,8 @@
# Arch-specific network modules
#
+KBUILD_CFLAGS += -O0
+
ifeq ($(CONFIG_X86_32),y)
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
else
Still see...
BROKEN: test_bpf: #294 BPF_MAXINSNS: Jump, gap, jump, ... jited:0
- Sedat -
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-27 7:36 ` Sedat Dilek
2019-07-27 8:16 ` Sedat Dilek
@ 2019-07-27 17:08 ` Yonghong Song
2019-07-28 11:09 ` Sedat Dilek
1 sibling, 1 reply; 15+ messages in thread
From: Yonghong Song @ 2019-07-27 17:08 UTC (permalink / raw)
To: sedat.dilek, Alexei Starovoitov
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On 7/27/19 12:36 AM, Sedat Dilek wrote:
> On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>>
>> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>
>>> On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
>>>>
>>>>
>>>>
>>>> On 7/26/19 2:02 PM, Sedat Dilek wrote:
>>>>> On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>>
>>>>>> Hi Yonghong Song,
>>>>>>
>>>>>> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
>>>>>>>
>>>>>>> Glad to know clang 9 has asm goto support and now It can compile
>>>>>>> kernel again.
>>>>>>>
>>>>>>
>>>>>> Yupp.
>>>>>>
>>>>>>>>
>>>>>>>> I am seeing a problem in the area bpf/seccomp causing
>>>>>>>> systemd/journald/udevd services to fail.
>>>>>>>>
>>>>>>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
>>>>>>>> to connect stdout to the journal socket, ignoring: Connection refused
>>>>>>>>
>>>>>>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
>>>>>>>> BFD linker ld.bfd on Debian/buster AMD64.
>>>>>>>> In both cases I use clang-9 (prerelease).
>>>>>>>
>>>>>>> Looks like it is a lld bug.
>>>>>>>
>>>>>>> I see the stack trace has __bpf_prog_run32() which is used by
>>>>>>> kernel bpf interpreter. Could you try to enable bpf jit
>>>>>>> sysctl net.core.bpf_jit_enable = 1
>>>>>>> If this passed, it will prove it is interpreter related.
>>>>>>>
>>>>>>
>>>>>> After...
>>>>>>
>>>>>> sysctl -w net.core.bpf_jit_enable=1
>>>>>>
>>>>>> I can start all failed systemd services.
>>>>>>
>>>>>> systemd-journald.service
>>>>>> systemd-udevd.service
>>>>>> haveged.service
>>>>>>
>>>>>> This is in maintenance mode.
>>>>>>
>>>>>> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
>>>>>>
>>>>>
>>>>> This is what I did:
>>>>
>>>> I probably won't have cycles to debug this potential lld issue.
>>>> Maybe you already did, I suggest you put enough reproducible
>>>> details in the bug you filed against lld so they can take a look.
>>>>
>>>
>>> I understand and will put the journalctl-log into the CBL issue
>>> tracker and update informations.
>>>
>>> Thanks for your help understanding the BPF correlations.
>>>
>>> Is setting 'net.core.bpf_jit_enable = 2' helpful here?
>>
>> jit_enable=1 is enough.
>> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
>>
>> It sounds like clang miscompiles interpreter.
>> modprobe test_bpf
>> should be able to point out which part of interpreter is broken.
>
> Maybe we need something like...
>
> "bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
>
> ...for clang?
Not sure how do you get conclusion it is gcse causing the problem.
But anyway, adding such flag in the kernel is not a good idea.
clang/llvm should be fixed instead. Esp. there is still time
for 9.0.0 release to fix bugs.
>
> - Sedat -
>
> [1] https://git.kernel.org/linus/3193c0836f203a91bef96d88c64cccf0be090d9c
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-27 8:16 ` Sedat Dilek
@ 2019-07-27 17:11 ` Yonghong Song
2019-07-28 11:16 ` Sedat Dilek
0 siblings, 1 reply; 15+ messages in thread
From: Yonghong Song @ 2019-07-27 17:11 UTC (permalink / raw)
To: sedat.dilek, Alexei Starovoitov
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On 7/27/19 1:16 AM, Sedat Dilek wrote:
> On Sat, Jul 27, 2019 at 9:36 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>>>
>>> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>
>>>> On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 7/26/19 2:02 PM, Sedat Dilek wrote:
>>>>>> On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Yonghong Song,
>>>>>>>
>>>>>>> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
>>>>>>>>
>>>>>>>> Glad to know clang 9 has asm goto support and now It can compile
>>>>>>>> kernel again.
>>>>>>>>
>>>>>>>
>>>>>>> Yupp.
>>>>>>>
>>>>>>>>>
>>>>>>>>> I am seeing a problem in the area bpf/seccomp causing
>>>>>>>>> systemd/journald/udevd services to fail.
>>>>>>>>>
>>>>>>>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
>>>>>>>>> to connect stdout to the journal socket, ignoring: Connection refused
>>>>>>>>>
>>>>>>>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
>>>>>>>>> BFD linker ld.bfd on Debian/buster AMD64.
>>>>>>>>> In both cases I use clang-9 (prerelease).
>>>>>>>>
>>>>>>>> Looks like it is a lld bug.
>>>>>>>>
>>>>>>>> I see the stack trace has __bpf_prog_run32() which is used by
>>>>>>>> kernel bpf interpreter. Could you try to enable bpf jit
>>>>>>>> sysctl net.core.bpf_jit_enable = 1
>>>>>>>> If this passed, it will prove it is interpreter related.
>>>>>>>>
>>>>>>>
>>>>>>> After...
>>>>>>>
>>>>>>> sysctl -w net.core.bpf_jit_enable=1
>>>>>>>
>>>>>>> I can start all failed systemd services.
>>>>>>>
>>>>>>> systemd-journald.service
>>>>>>> systemd-udevd.service
>>>>>>> haveged.service
>>>>>>>
>>>>>>> This is in maintenance mode.
>>>>>>>
>>>>>>> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
>>>>>>>
>>>>>>
>>>>>> This is what I did:
>>>>>
>>>>> I probably won't have cycles to debug this potential lld issue.
>>>>> Maybe you already did, I suggest you put enough reproducible
>>>>> details in the bug you filed against lld so they can take a look.
>>>>>
>>>>
>>>> I understand and will put the journalctl-log into the CBL issue
>>>> tracker and update informations.
>>>>
>>>> Thanks for your help understanding the BPF correlations.
>>>>
>>>> Is setting 'net.core.bpf_jit_enable = 2' helpful here?
>>>
>>> jit_enable=1 is enough.
>>> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
>>>
>>> It sounds like clang miscompiles interpreter.
>
> Just to clarify:
> This does not happen with clang-9 + ld.bfd (GNU/ld linker).
>
>>> modprobe test_bpf
>>> should be able to point out which part of interpreter is broken.
>>
>> Maybe we need something like...
>>
>> "bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
>>
>> ...for clang?
>>
>
> Not sure if something like GCC's...
>
> -fgcse
>
> Perform a global common subexpression elimination pass. This pass also
> performs global constant and copy propagation.
>
> Note: When compiling a program using computed gotos, a GCC extension,
> you may get better run-time performance if you disable the global
> common subexpression elimination pass by adding -fno-gcse to the
> command line.
>
> Enabled at levels -O2, -O3, -Os.
>
> ...is available for clang.
>
> I tried with hopping to turn off "global common subexpression elimination":
>
> diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
> index 383c87300b0d..92f934a1e9ff 100644
> --- a/arch/x86/net/Makefile
> +++ b/arch/x86/net/Makefile
> @@ -3,6 +3,8 @@
> # Arch-specific network modules
> #
>
> +KBUILD_CFLAGS += -O0
This won't work. First, you added to the wrong file. The interpreter
is at kernel/bpf/core.c.
Second, kernel may have compilation issues with -O0.
> +
> ifeq ($(CONFIG_X86_32),y)
> obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
> else
>
> Still see...
> BROKEN: test_bpf: #294 BPF_MAXINSNS: Jump, gap, jump, ... jited:0
>
> - Sedat -
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-27 17:08 ` Yonghong Song
@ 2019-07-28 11:09 ` Sedat Dilek
0 siblings, 0 replies; 15+ messages in thread
From: Sedat Dilek @ 2019-07-28 11:09 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Martin Lau, Song Liu, netdev, bpf, Clang-Built-Linux ML,
Kees Cook, Nick Desaulniers, Nathan Chancellor
On Sat, Jul 27, 2019 at 7:08 PM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 7/27/19 12:36 AM, Sedat Dilek wrote:
> > On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> >>
> >> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>
> >>> On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 7/26/19 2:02 PM, Sedat Dilek wrote:
> >>>>> On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi Yonghong Song,
> >>>>>>
> >>>>>> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> >>>>>>>
> >>>>>>> Glad to know clang 9 has asm goto support and now It can compile
> >>>>>>> kernel again.
> >>>>>>>
> >>>>>>
> >>>>>> Yupp.
> >>>>>>
> >>>>>>>>
> >>>>>>>> I am seeing a problem in the area bpf/seccomp causing
> >>>>>>>> systemd/journald/udevd services to fail.
> >>>>>>>>
> >>>>>>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> >>>>>>>> to connect stdout to the journal socket, ignoring: Connection refused
> >>>>>>>>
> >>>>>>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> >>>>>>>> BFD linker ld.bfd on Debian/buster AMD64.
> >>>>>>>> In both cases I use clang-9 (prerelease).
> >>>>>>>
> >>>>>>> Looks like it is a lld bug.
> >>>>>>>
> >>>>>>> I see the stack trace has __bpf_prog_run32() which is used by
> >>>>>>> kernel bpf interpreter. Could you try to enable bpf jit
> >>>>>>> sysctl net.core.bpf_jit_enable = 1
> >>>>>>> If this passed, it will prove it is interpreter related.
> >>>>>>>
> >>>>>>
> >>>>>> After...
> >>>>>>
> >>>>>> sysctl -w net.core.bpf_jit_enable=1
> >>>>>>
> >>>>>> I can start all failed systemd services.
> >>>>>>
> >>>>>> systemd-journald.service
> >>>>>> systemd-udevd.service
> >>>>>> haveged.service
> >>>>>>
> >>>>>> This is in maintenance mode.
> >>>>>>
> >>>>>> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> >>>>>>
> >>>>>
> >>>>> This is what I did:
> >>>>
> >>>> I probably won't have cycles to debug this potential lld issue.
> >>>> Maybe you already did, I suggest you put enough reproducible
> >>>> details in the bug you filed against lld so they can take a look.
> >>>>
> >>>
> >>> I understand and will put the journalctl-log into the CBL issue
> >>> tracker and update informations.
> >>>
> >>> Thanks for your help understanding the BPF correlations.
> >>>
> >>> Is setting 'net.core.bpf_jit_enable = 2' helpful here?
> >>
> >> jit_enable=1 is enough.
> >> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
> >>
> >> It sounds like clang miscompiles interpreter.
> >> modprobe test_bpf
> >> should be able to point out which part of interpreter is broken.
> >
> > Maybe we need something like...
> >
> > "bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
> >
> > ...for clang?
>
> Not sure how do you get conclusion it is gcse causing the problem.
> But anyway, adding such flag in the kernel is not a good idea.
> clang/llvm should be fixed instead. Esp. there is still time
> for 9.0.0 release to fix bugs.
>
To clarify: This is a snapshot release of clang-9 built with tc-build.
Building with -O0 is not possible as I see asm-goto failing.
- Sedat -
[1] https://github.com/ClangBuiltLinux/tc-build
> >
> > - Sedat -
> >
> > [1] https://git.kernel.org/linus/3193c0836f203a91bef96d88c64cccf0be090d9c
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-27 17:11 ` Yonghong Song
@ 2019-07-28 11:16 ` Sedat Dilek
0 siblings, 0 replies; 15+ messages in thread
From: Sedat Dilek @ 2019-07-28 11:16 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Martin Lau, Song Liu, netdev, bpf, Clang-Built-Linux ML,
Kees Cook, Nick Desaulniers, Nathan Chancellor
On Sat, Jul 27, 2019 at 7:11 PM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 7/27/19 1:16 AM, Sedat Dilek wrote:
> > On Sat, Jul 27, 2019 at 9:36 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>
> >> On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
> >> <alexei.starovoitov@gmail.com> wrote:
> >>>
> >>> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>>
> >>>> On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 7/26/19 2:02 PM, Sedat Dilek wrote:
> >>>>>> On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi Yonghong Song,
> >>>>>>>
> >>>>>>> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> >>>>>>>>
> >>>>>>>> Glad to know clang 9 has asm goto support and now It can compile
> >>>>>>>> kernel again.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yupp.
> >>>>>>>
> >>>>>>>>>
> >>>>>>>>> I am seeing a problem in the area bpf/seccomp causing
> >>>>>>>>> systemd/journald/udevd services to fail.
> >>>>>>>>>
> >>>>>>>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> >>>>>>>>> to connect stdout to the journal socket, ignoring: Connection refused
> >>>>>>>>>
> >>>>>>>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> >>>>>>>>> BFD linker ld.bfd on Debian/buster AMD64.
> >>>>>>>>> In both cases I use clang-9 (prerelease).
> >>>>>>>>
> >>>>>>>> Looks like it is a lld bug.
> >>>>>>>>
> >>>>>>>> I see the stack trace has __bpf_prog_run32() which is used by
> >>>>>>>> kernel bpf interpreter. Could you try to enable bpf jit
> >>>>>>>> sysctl net.core.bpf_jit_enable = 1
> >>>>>>>> If this passed, it will prove it is interpreter related.
> >>>>>>>>
> >>>>>>>
> >>>>>>> After...
> >>>>>>>
> >>>>>>> sysctl -w net.core.bpf_jit_enable=1
> >>>>>>>
> >>>>>>> I can start all failed systemd services.
> >>>>>>>
> >>>>>>> systemd-journald.service
> >>>>>>> systemd-udevd.service
> >>>>>>> haveged.service
> >>>>>>>
> >>>>>>> This is in maintenance mode.
> >>>>>>>
> >>>>>>> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> >>>>>>>
> >>>>>>
> >>>>>> This is what I did:
> >>>>>
> >>>>> I probably won't have cycles to debug this potential lld issue.
> >>>>> Maybe you already did, I suggest you put enough reproducible
> >>>>> details in the bug you filed against lld so they can take a look.
> >>>>>
> >>>>
> >>>> I understand and will put the journalctl-log into the CBL issue
> >>>> tracker and update informations.
> >>>>
> >>>> Thanks for your help understanding the BPF correlations.
> >>>>
> >>>> Is setting 'net.core.bpf_jit_enable = 2' helpful here?
> >>>
> >>> jit_enable=1 is enough.
> >>> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
> >>>
> >>> It sounds like clang miscompiles interpreter.
> >
> > Just to clarify:
> > This does not happen with clang-9 + ld.bfd (GNU/ld linker).
> >
> >>> modprobe test_bpf
> >>> should be able to point out which part of interpreter is broken.
> >>
> >> Maybe we need something like...
> >>
> >> "bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
> >>
> >> ...for clang?
> >>
> >
> > Not sure if something like GCC's...
> >
> > -fgcse
> >
> > Perform a global common subexpression elimination pass. This pass also
> > performs global constant and copy propagation.
> >
> > Note: When compiling a program using computed gotos, a GCC extension,
> > you may get better run-time performance if you disable the global
> > common subexpression elimination pass by adding -fno-gcse to the
> > command line.
> >
> > Enabled at levels -O2, -O3, -Os.
> >
> > ...is available for clang.
> >
> > I tried with hopping to turn off "global common subexpression elimination":
> >
> > diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
> > index 383c87300b0d..92f934a1e9ff 100644
> > --- a/arch/x86/net/Makefile
> > +++ b/arch/x86/net/Makefile
> > @@ -3,6 +3,8 @@
> > # Arch-specific network modules
> > #
> >
> > +KBUILD_CFLAGS += -O0
>
> This won't work. First, you added to the wrong file. The interpreter
> is at kernel/bpf/core.c.
>
Thanks for the clarification.
I mixed up the x86 BPF JIT compiler with the BPF interpreter.
I see no diff in the disassembled kernel/bpf/core.o in my clang9-bfd
and clang9-lld build-dirs.
l$ objdump -M intel -d linux.clang9-bfd/kernel/bpf/core.o >
bpf_core_o_clang9-bfd.txt
$ objdump -M intel -d linux.clang9-lld/kernel/bpf/core.o >
bpf_core_o_clang9-lld.txt
--- bpf_core_o_clang9-bfd.txt 2019-07-28 13:11:59.363552042 +0200
+++ bpf_core_o_clang9-lld.txt 2019-07-28 13:12:09.975535278 +0200
@@ -1,5 +1,5 @@
-linux.clang9-bfd/kernel/bpf/core.o: file format elf64-x86-64
+linux.clang9-lld/kernel/bpf/core.o: file format elf64-x86-64
Disassembly of section .text:
> Second, kernel may have compilation issues with -O0.
>
Confirmed.
- Sedat -
> > +
> > ifeq ($(CONFIG_X86_32),y)
> > obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
> > else
> >
> > Still see...
> > BROKEN: test_bpf: #294 BPF_MAXINSNS: Jump, gap, jump, ... jited:0
> >
> > - Sedat -
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-07-26 15:45 ` next-20190723: bpf/seccomp - systemd/journald issue? Yonghong Song
2019-07-26 20:38 ` Sedat Dilek
@ 2019-08-01 7:39 ` Sedat Dilek
2019-08-01 9:35 ` Sedat Dilek
1 sibling, 1 reply; 15+ messages in thread
From: Sedat Dilek @ 2019-08-01 7:39 UTC (permalink / raw)
To: Yonghong Song, Josh Poimboeuf
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
Hi,
just want to let you know that I did a git bisect with Linux v5.3-rc2
(where the problem also exists) and the result (details see [1]):
e55a73251da335873a6e87d68fb17e5aabb8978e is the first bad commit
commit e55a73251da335873a6e87d68fb17e5aabb8978e
Author: Josh Poimboeuf <jpoimboe@redhat.com>
Date: Thu Jun 27 20:50:47 2019 -0500
bpf: Fix ORC unwinding in non-JIT BPF code
Objtool previously ignored ___bpf_prog_run() because it didn't understand
the jump table. This resulted in the ORC unwinder not being able to unwind
through non-JIT BPF code.
Now that objtool knows how to read jump tables, remove the whitelist and
annotate the jump table so objtool can recognize it.
Also add an additional "const" to the jump table definition to clarify that
the text pointers are constant. Otherwise GCC sets the section writable
flag and the assembler spits out warnings.
Fixes: d15d356887e7 ("perf/x86: Make perf callchains work without
CONFIG_FRAME_POINTER")
Reported-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Kairui Song <kasong@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lkml.kernel.org/r/881939122b88f32be4c374d248c09d7527a87e35.1561685471.git.jpoimboe@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
:040000 040000 4735e9d14fa416c1c361ec3923440a3d586a627d
31de80b85c7b0292e47a719ecb6b1a451de2f8ef M kernel
Maybe you want to look at this, too.
The object files are attached in [2].
Thanks,
- Sedat -
[0] https://github.com/ClangBuiltLinux/linux/issues/619
[1] https://github.com/ClangBuiltLinux/linux/issues/619#issuecomment-517152467
[2] https://github.com/ClangBuiltLinux/linux/issues/619#issuecomment-517159635
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: next-20190723: bpf/seccomp - systemd/journald issue?
2019-08-01 7:39 ` Sedat Dilek
@ 2019-08-01 9:35 ` Sedat Dilek
0 siblings, 0 replies; 15+ messages in thread
From: Sedat Dilek @ 2019-08-01 9:35 UTC (permalink / raw)
To: Yonghong Song, Josh Poimboeuf
Cc: Alexei Starovoitov, Daniel Borkmann, Martin Lau, Song Liu,
netdev, bpf, Clang-Built-Linux ML, Kees Cook, Nick Desaulniers,
Nathan Chancellor
On Thu, Aug 1, 2019 at 9:39 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> Hi,
>
> just want to let you know that I did a git bisect with Linux v5.3-rc2
> (where the problem also exists) and the result (details see [1]):
>
> e55a73251da335873a6e87d68fb17e5aabb8978e is the first bad commit
> commit e55a73251da335873a6e87d68fb17e5aabb8978e
> Author: Josh Poimboeuf <jpoimboe@redhat.com>
> Date: Thu Jun 27 20:50:47 2019 -0500
>
> bpf: Fix ORC unwinding in non-JIT BPF code
>
> Objtool previously ignored ___bpf_prog_run() because it didn't understand
> the jump table. This resulted in the ORC unwinder not being able to unwind
> through non-JIT BPF code.
>
> Now that objtool knows how to read jump tables, remove the whitelist and
> annotate the jump table so objtool can recognize it.
>
> Also add an additional "const" to the jump table definition to clarify that
> the text pointers are constant. Otherwise GCC sets the section writable
> flag and the assembler spits out warnings.
>
> Fixes: d15d356887e7 ("perf/x86: Make perf callchains work without
> CONFIG_FRAME_POINTER")
> Reported-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Acked-by: Alexei Starovoitov <ast@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Kairui Song <kasong@redhat.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Daniel Borkmann <daniel@iogearbox.net>
> Link: https://lkml.kernel.org/r/881939122b88f32be4c374d248c09d7527a87e35.1561685471.git.jpoimboe@redhat.com
> Signed-off-by: Ingo Molnar <mingo@kernel.org>
>
> :040000 040000 4735e9d14fa416c1c361ec3923440a3d586a627d
> 31de80b85c7b0292e47a719ecb6b1a451de2f8ef M kernel
>
> Maybe you want to look at this, too.
>
> The object files are attached in [2].
>
> Thanks,
> - Sedat -
>
> [0] https://github.com/ClangBuiltLinux/linux/issues/619
> [1] https://github.com/ClangBuiltLinux/linux/issues/619#issuecomment-517152467
> [2] https://github.com/ClangBuiltLinux/linux/issues/619#issuecomment-517159635
After reverting above commit I can boot into Linux v5.3-rc2 built with
clang-9.0.0-rc1 and lld with no issues.
- Sedat -
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-08-01 9:36 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CA+icZUWF=B_phP8eGD3v2d9jSSK6Y-N65y-T6xewZnY91vc2_Q@mail.gmail.com>
2019-07-26 15:45 ` next-20190723: bpf/seccomp - systemd/journald issue? Yonghong Song
2019-07-26 20:38 ` Sedat Dilek
2019-07-26 21:02 ` Sedat Dilek
2019-07-26 21:10 ` Yonghong Song
2019-07-26 21:19 ` Sedat Dilek
2019-07-27 2:24 ` Alexei Starovoitov
2019-07-27 7:36 ` Sedat Dilek
2019-07-27 8:16 ` Sedat Dilek
2019-07-27 17:11 ` Yonghong Song
2019-07-28 11:16 ` Sedat Dilek
2019-07-27 17:08 ` Yonghong Song
2019-07-28 11:09 ` Sedat Dilek
2019-07-26 21:05 ` Yonghong Song
2019-08-01 7:39 ` Sedat Dilek
2019-08-01 9:35 ` Sedat Dilek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).