pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

Hi,

I also noticed that after upgrading pahole to v1.24 kernel build (tested on
v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:

    BTFIDS  vmlinux
  + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
  FAILED: load BTF from vmlinux: Invalid argument

Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
v1.23 resolves the issue.

Thanks,

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

Arnaldo,

On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> >
> > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> >
> >     BTFIDS  vmlinux
> >   + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> >   FAILED: load BTF from vmlinux: Invalid argument
> >
> > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > v1.23 resolves the issue.
> >
> 
> Can you try this, from Martin Reboredo (Archlinux):
> 
> Can you try a build of the kernel or the by passing the
> --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> 
> Here's a patch for either in tree scripts/pahole-flags.sh or
> /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh

This patch helped and kernel builds successfully after applying it.
(Didn't notice this suggestion in release discussion thread.)

Thanks!

> 
> diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> index 0d99ef17e4a528..1f1f1d397c399a 100755
> --- a/scripts/pahole-flags.sh
> +++ b/scripts/pahole-flags.sh
> @@ -19,5 +19,9 @@ fi
>  if [ "${pahole_ver}" -ge "122" ]; then
>         extra_paholeopt="${extra_paholeopt} -j"
>  fi
> +if [ "${pahole_ver}" -ge "124" ]; then
> +       # see PAHOLE_HAS_LANG_EXCLUDE
> +       extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_enum64"
> +fi
> 
>  echo ${extra_paholeopt}
> 
> >
> > Thanks,
> >
> >

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

Arnaldo,

On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > >
> > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > >
> > >     BTFIDS  vmlinux
> > >   + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > >   FAILED: load BTF from vmlinux: Invalid argument
> > >
> > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > v1.23 resolves the issue.
> > >
> > 
> > Can you try this, from Martin Reboredo (Archlinux):
> > 
> > Can you try a build of the kernel or the by passing the
> > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > 
> > Here's a patch for either in tree scripts/pahole-flags.sh or
> > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> 
> This patch helped and kernel builds successfully after applying it.
> (Didn't notice this suggestion in release discussion thread.)

Even thought it now compiles with this patch, it does not boot
afterwards (in virtme-like env), witch such console messages:

  [    0.767649] Run /init as init process
  [    0.770858] BPF:[593] ENUM perf_event_task_context
  [    0.771262] BPF:size=4 vlen=4
  [    0.771511] BPF:
  [    0.771680] BPF:Invalid btf_info kind_flag
  [    0.772016] BPF:
  [    0.772016]
  [    0.772288] failed to validate module [9pnet] BTF: -22
  init_module '9pnet.ko' error -1
  [    0.785515] 9p: Unknown symbol p9_client_getattr_dotl (err -2)
  [    0.786005] 9p: Unknown symbol p9_client_wstat (err -2)
  [    0.786438] 9p: Unknown symbol p9_client_open (err -2)
  [    0.786863] 9p: Unknown symbol p9_client_rename (err -2)
  [    0.787307] 9p: Unknown symbol p9_client_remove (err -2)
  [    0.787749] 9p: Unknown symbol p9_client_renameat (err -2)
  [    0.788202] 9p: Unknown symbol p9_client_fcreate (err -2)
  [    0.788651] 9p: Unknown symbol p9_is_proto_dotu (err -2)
  [    0.789086] 9p: Unknown symbol p9_client_disconnect (err -2)
  [    0.789558] 9p: Unknown symbol p9_client_attach (err -2)
  [    0.790001] 9p: Unknown symbol p9stat_free (err -2)
  [    0.790409] 9p: Unknown symbol p9_client_create (err -2)
  [    0.790856] 9p: Unknown symbol p9_client_read_once (err -2)
  [    0.791314] 9p: Unknown symbol p9_client_setattr (err -2)
  [    0.791762] 9p: Unknown symbol p9_client_xattrwalk (err -2)
  [    0.792222] 9p: Unknown symbol p9_client_destroy (err -2)
  [    0.792674] 9p: Unknown symbol p9_client_unlinkat (err -2)
  [    0.793124] 9p: Unknown symbol p9_client_mkdir_dotl (err -2)
  [    0.793585] 9p: Unknown symbol p9_client_xattrcreate (err -2)
  [    0.794055] 9p: Unknown symbol p9_client_create_dotl (err -2)
  [    0.794527] 9p: Unknown symbol p9_client_lock_dotl (err -2)
  [    0.794985] 9p: Unknown symbol p9_client_write (err -2)
  [    0.795415] 9p: Unknown symbol p9_client_walk (err -2)
  [    0.795837] 9p: Unknown symbol p9_show_client_options (err -2)
  [    0.796308] 9p: Unknown symbol p9_client_read (err -2)
  [    0.796731] 9p: Unknown symbol p9_client_fsync (err -2)
  [    0.797158] 9p: Unknown symbol p9dirent_read (err -2)
  [    0.797572] 9p: Unknown symbol p9_client_symlink (err -2)
  [    0.798016] 9p: Unknown symbol p9_client_readlink (err -2)
  [    0.798470] 9p: Unknown symbol p9_is_proto_dotl (err -2)
  [    0.798908] 9p: Unknown symbol p9_client_clunk (err -2)
  [    0.799339] 9p: Unknown symbol p9stat_read (err -2)
  [    0.799747] 9p: Unknown symbol p9_client_statfs (err -2)
  [    0.800184] 9p: Unknown symbol p9_client_link (err -2)
  [    0.800612] 9p: Unknown symbol p9_client_stat (err -2)
  [    0.801038] 9p: Unknown symbol p9_client_begin_disconnect (err -2)
  [    0.801544] 9p: Unknown symbol p9_client_getlock_dotl (err -2)
  [    0.802024] 9p: Unknown symbol p9_client_readdir (err -2)
  [    0.802469] 9p: Unknown symbol p9_client_mknod_dotl (err -2)
  init_module '9p.ko' error -1
  [    0.809193] failed to validate module [virtio] BTF: -22
  init_module 'virtio.ko' error -1
  [    0.825316] failed to validate module [virtio_ring] BTF: -22
  init_module 'virtio_ring.ko' error -1
  [    0.841110] 9pnet_virtio: Unknown symbol register_virtio_driver (err -2)
  [    0.841674] 9pnet_virtio: Unknown symbol p9_req_put (err -2)
  [    0.842143] 9pnet_virtio: Unknown symbol unregister_virtio_driver (err -2)
  [    0.842708] 9pnet_virtio: Unknown symbol p9_release_pages (err -2)
  [    0.843209] 9pnet_virtio: Unknown symbol virtqueue_add_sgs (err -2)
  [    0.843728] 9pnet_virtio: Unknown symbol virtqueue_get_buf (err -2)
  [    0.844237] 9pnet_virtio: Unknown symbol virtqueue_kick (err -2)
  [    0.844733] 9pnet_virtio: Unknown symbol v9fs_register_trans (err -2)
  [    0.845256] 9pnet_virtio: Unknown symbol virtio_check_driver_offered_feature (err -2)
  [    0.845893] 9pnet_virtio: Unknown symbol v9fs_unregister_trans (err -2)
  [    0.846434] 9pnet_virtio: Unknown symbol p9_client_cb (err -2)
  init_module '9pnet_virtio.ko' error -1
  [    0.853175] failed to validate module [virtio_pci_modern_dev] BTF: -22
  init_module 'virtio_pci_modern_dev.ko' error -1
  [    0.869196] virtio_pci: Unknown symbol vring_transport_features (err -2)
  [    0.869759] virtio_pci: Unknown symbol vp_modern_get_status (err -2)
  [    0.870276] virtio_pci: Unknown symbol vp_modern_map_vq_notify (err -2)
  [    0.870817] virtio_pci: Unknown symbol virtqueue_get_avail_addr (err -2)
  [    0.871363] virtio_pci: Unknown symbol vp_modern_get_queue_size (err -2)
  [    0.871912] virtio_pci: Unknown symbol vp_modern_remove (err -2)
  [    0.872405] virtio_pci: Unknown symbol vring_create_virtqueue (err -2)
  [    0.872938] virtio_pci: Unknown symbol vp_modern_get_features (err -2)
  [    0.873470] virtio_pci: Unknown symbol vp_modern_set_status (err -2)
  [    0.873986] virtio_pci: Unknown symbol vp_modern_queue_address (err -2)
  [    0.874523] virtio_pci: Unknown symbol virtio_device_restore (err -2)
  [    0.875050] virtio_pci: Unknown symbol vring_interrupt (err -2)
  [    0.875544] virtio_pci: Unknown symbol virtio_config_changed (err -2)
  [    0.876073] virtio_pci: Unknown symbol vp_modern_get_queue_enable (err -2)
  [    0.876639] virtio_pci: Unknown symbol vp_modern_config_vector (err -2)
  [    0.877172] virtio_pci: Unknown symbol virtqueue_get_vring_size (err -2)
  [    0.877721] virtio_pci: Unknown symbol vp_modern_probe (err -2)
  [    0.878205] virtio_pci: Unknown symbol unregister_virtio_device (err -2)
  [    0.878753] virtio_pci: Unknown symbol virtqueue_get_desc_addr (err -2)
  [    0.879288] virtio_pci: Unknown symbol vp_modern_set_queue_size (err -2)
  [    0.879836] virtio_pci: Unknown symbol vp_modern_queue_vector (err -2)
  [    0.880371] virtio_pci: Unknown symbol virtio_device_freeze (err -2)
  [    0.880896] virtio_pci: Unknown symbol vp_modern_generation (err -2)
  [    0.881417] virtio_pci: Unknown symbol virtio_break_device (err -2)
  [    0.881969] virtio_pci: Unknown symbol vp_modern_set_features (err -2)
  [    0.882501] virtio_pci: Unknown symbol virtqueue_get_used_addr (err -2)
  [    0.883014] virtio_pci: Unknown symbol register_virtio_device (err -2)
  [    0.883547] virtio_pci: Unknown symbol vp_modern_get_num_queues (err -2)
  [    0.884090] virtio_pci: Unknown symbol vp_modern_set_queue_enable (err -2)
  [    0.884684] virtio_pci: Unknown symbol vring_del_virtqueue (err -2)
  init_module 'virtio_pci.ko' error -1

It seems the same is happened in Arch:

  https://bbs.archlinux.org/viewtopic.php?id=279132

And they reverted pahole to 1.23:

  https://github.com/archlinux/svntogit-packages/commits/packages/pahole/trunk/PKGBUILD

When I'm revering dwarves to 1.23 it resolves the issue for me too (in
ALT).

Thanks,


> 
> Thanks!
> 
> > 
> > diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> > index 0d99ef17e4a528..1f1f1d397c399a 100755
> > --- a/scripts/pahole-flags.sh
> > +++ b/scripts/pahole-flags.sh
> > @@ -19,5 +19,9 @@ fi
> >  if [ "${pahole_ver}" -ge "122" ]; then
> >         extra_paholeopt="${extra_paholeopt} -j"
> >  fi
> > +if [ "${pahole_ver}" -ge "124" ]; then
> > +       # see PAHOLE_HAS_LANG_EXCLUDE
> > +       extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_enum64"
> > +fi
> > 
> >  echo ${extra_paholeopt}
> > 
> > >
> > > Thanks,
> > >
> > >

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> Arnaldo,
> 
> On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > >
> > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > >
> > > >     BTFIDS  vmlinux
> > > >   + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > >   FAILED: load BTF from vmlinux: Invalid argument
> > > >
> > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > v1.23 resolves the issue.
> > > >
> > > 
> > > Can you try this, from Martin Reboredo (Archlinux):
> > > 
> > > Can you try a build of the kernel or the by passing the
> > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > 
> > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > 
> > This patch helped and kernel builds successfully after applying it.
> > (Didn't notice this suggestion in release discussion thread.)
> 
> Even thought it now compiles with this patch, it does not boot
> afterwards (in virtme-like env), witch such console messages:

I'm talking here about 5.15.62. Yes, proposed patch does not apply there
(since there is no `scripts/pahole-flags.sh`), but I updated
`scripts/link-vmlinux.sh` with the similar `if` to append
`--skip_encoding_btf_enum64` which lets then compilation pass.

Thanks,

> 
>   [    0.767649] Run /init as init process
>   [    0.770858] BPF:[593] ENUM perf_event_task_context
>   [    0.771262] BPF:size=4 vlen=4
>   [    0.771511] BPF:
>   [    0.771680] BPF:Invalid btf_info kind_flag
>   [    0.772016] BPF:
>   [    0.772016]
>   [    0.772288] failed to validate module [9pnet] BTF: -22
>   init_module '9pnet.ko' error -1
>   [    0.785515] 9p: Unknown symbol p9_client_getattr_dotl (err -2)
>   [    0.786005] 9p: Unknown symbol p9_client_wstat (err -2)
>   [    0.786438] 9p: Unknown symbol p9_client_open (err -2)
>   [    0.786863] 9p: Unknown symbol p9_client_rename (err -2)
>   [    0.787307] 9p: Unknown symbol p9_client_remove (err -2)
>   [    0.787749] 9p: Unknown symbol p9_client_renameat (err -2)
>   [    0.788202] 9p: Unknown symbol p9_client_fcreate (err -2)
>   [    0.788651] 9p: Unknown symbol p9_is_proto_dotu (err -2)
>   [    0.789086] 9p: Unknown symbol p9_client_disconnect (err -2)
>   [    0.789558] 9p: Unknown symbol p9_client_attach (err -2)
>   [    0.790001] 9p: Unknown symbol p9stat_free (err -2)
>   [    0.790409] 9p: Unknown symbol p9_client_create (err -2)
>   [    0.790856] 9p: Unknown symbol p9_client_read_once (err -2)
>   [    0.791314] 9p: Unknown symbol p9_client_setattr (err -2)
>   [    0.791762] 9p: Unknown symbol p9_client_xattrwalk (err -2)
>   [    0.792222] 9p: Unknown symbol p9_client_destroy (err -2)
>   [    0.792674] 9p: Unknown symbol p9_client_unlinkat (err -2)
>   [    0.793124] 9p: Unknown symbol p9_client_mkdir_dotl (err -2)
>   [    0.793585] 9p: Unknown symbol p9_client_xattrcreate (err -2)
>   [    0.794055] 9p: Unknown symbol p9_client_create_dotl (err -2)
>   [    0.794527] 9p: Unknown symbol p9_client_lock_dotl (err -2)
>   [    0.794985] 9p: Unknown symbol p9_client_write (err -2)
>   [    0.795415] 9p: Unknown symbol p9_client_walk (err -2)
>   [    0.795837] 9p: Unknown symbol p9_show_client_options (err -2)
>   [    0.796308] 9p: Unknown symbol p9_client_read (err -2)
>   [    0.796731] 9p: Unknown symbol p9_client_fsync (err -2)
>   [    0.797158] 9p: Unknown symbol p9dirent_read (err -2)
>   [    0.797572] 9p: Unknown symbol p9_client_symlink (err -2)
>   [    0.798016] 9p: Unknown symbol p9_client_readlink (err -2)
>   [    0.798470] 9p: Unknown symbol p9_is_proto_dotl (err -2)
>   [    0.798908] 9p: Unknown symbol p9_client_clunk (err -2)
>   [    0.799339] 9p: Unknown symbol p9stat_read (err -2)
>   [    0.799747] 9p: Unknown symbol p9_client_statfs (err -2)
>   [    0.800184] 9p: Unknown symbol p9_client_link (err -2)
>   [    0.800612] 9p: Unknown symbol p9_client_stat (err -2)
>   [    0.801038] 9p: Unknown symbol p9_client_begin_disconnect (err -2)
>   [    0.801544] 9p: Unknown symbol p9_client_getlock_dotl (err -2)
>   [    0.802024] 9p: Unknown symbol p9_client_readdir (err -2)
>   [    0.802469] 9p: Unknown symbol p9_client_mknod_dotl (err -2)
>   init_module '9p.ko' error -1
>   [    0.809193] failed to validate module [virtio] BTF: -22
>   init_module 'virtio.ko' error -1
>   [    0.825316] failed to validate module [virtio_ring] BTF: -22
>   init_module 'virtio_ring.ko' error -1
>   [    0.841110] 9pnet_virtio: Unknown symbol register_virtio_driver (err -2)
>   [    0.841674] 9pnet_virtio: Unknown symbol p9_req_put (err -2)
>   [    0.842143] 9pnet_virtio: Unknown symbol unregister_virtio_driver (err -2)
>   [    0.842708] 9pnet_virtio: Unknown symbol p9_release_pages (err -2)
>   [    0.843209] 9pnet_virtio: Unknown symbol virtqueue_add_sgs (err -2)
>   [    0.843728] 9pnet_virtio: Unknown symbol virtqueue_get_buf (err -2)
>   [    0.844237] 9pnet_virtio: Unknown symbol virtqueue_kick (err -2)
>   [    0.844733] 9pnet_virtio: Unknown symbol v9fs_register_trans (err -2)
>   [    0.845256] 9pnet_virtio: Unknown symbol virtio_check_driver_offered_feature (err -2)
>   [    0.845893] 9pnet_virtio: Unknown symbol v9fs_unregister_trans (err -2)
>   [    0.846434] 9pnet_virtio: Unknown symbol p9_client_cb (err -2)
>   init_module '9pnet_virtio.ko' error -1
>   [    0.853175] failed to validate module [virtio_pci_modern_dev] BTF: -22
>   init_module 'virtio_pci_modern_dev.ko' error -1
>   [    0.869196] virtio_pci: Unknown symbol vring_transport_features (err -2)
>   [    0.869759] virtio_pci: Unknown symbol vp_modern_get_status (err -2)
>   [    0.870276] virtio_pci: Unknown symbol vp_modern_map_vq_notify (err -2)
>   [    0.870817] virtio_pci: Unknown symbol virtqueue_get_avail_addr (err -2)
>   [    0.871363] virtio_pci: Unknown symbol vp_modern_get_queue_size (err -2)
>   [    0.871912] virtio_pci: Unknown symbol vp_modern_remove (err -2)
>   [    0.872405] virtio_pci: Unknown symbol vring_create_virtqueue (err -2)
>   [    0.872938] virtio_pci: Unknown symbol vp_modern_get_features (err -2)
>   [    0.873470] virtio_pci: Unknown symbol vp_modern_set_status (err -2)
>   [    0.873986] virtio_pci: Unknown symbol vp_modern_queue_address (err -2)
>   [    0.874523] virtio_pci: Unknown symbol virtio_device_restore (err -2)
>   [    0.875050] virtio_pci: Unknown symbol vring_interrupt (err -2)
>   [    0.875544] virtio_pci: Unknown symbol virtio_config_changed (err -2)
>   [    0.876073] virtio_pci: Unknown symbol vp_modern_get_queue_enable (err -2)
>   [    0.876639] virtio_pci: Unknown symbol vp_modern_config_vector (err -2)
>   [    0.877172] virtio_pci: Unknown symbol virtqueue_get_vring_size (err -2)
>   [    0.877721] virtio_pci: Unknown symbol vp_modern_probe (err -2)
>   [    0.878205] virtio_pci: Unknown symbol unregister_virtio_device (err -2)
>   [    0.878753] virtio_pci: Unknown symbol virtqueue_get_desc_addr (err -2)
>   [    0.879288] virtio_pci: Unknown symbol vp_modern_set_queue_size (err -2)
>   [    0.879836] virtio_pci: Unknown symbol vp_modern_queue_vector (err -2)
>   [    0.880371] virtio_pci: Unknown symbol virtio_device_freeze (err -2)
>   [    0.880896] virtio_pci: Unknown symbol vp_modern_generation (err -2)
>   [    0.881417] virtio_pci: Unknown symbol virtio_break_device (err -2)
>   [    0.881969] virtio_pci: Unknown symbol vp_modern_set_features (err -2)
>   [    0.882501] virtio_pci: Unknown symbol virtqueue_get_used_addr (err -2)
>   [    0.883014] virtio_pci: Unknown symbol register_virtio_device (err -2)
>   [    0.883547] virtio_pci: Unknown symbol vp_modern_get_num_queues (err -2)
>   [    0.884090] virtio_pci: Unknown symbol vp_modern_set_queue_enable (err -2)
>   [    0.884684] virtio_pci: Unknown symbol vring_del_virtqueue (err -2)
>   init_module 'virtio_pci.ko' error -1
> 
> It seems the same is happened in Arch:
> 
>   https://bbs.archlinux.org/viewtopic.php?id=279132
> 
> And they reverted pahole to 1.23:
> 
>   https://github.com/archlinux/svntogit-packages/commits/packages/pahole/trunk/PKGBUILD
> 
> When I'm revering dwarves to 1.23 it resolves the issue for me too (in
> ALT).
> 
> Thanks,
> 
> 
> > 
> > Thanks!
> > 
> > > 
> > > diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> > > index 0d99ef17e4a528..1f1f1d397c399a 100755
> > > --- a/scripts/pahole-flags.sh
> > > +++ b/scripts/pahole-flags.sh
> > > @@ -19,5 +19,9 @@ fi
> > >  if [ "${pahole_ver}" -ge "122" ]; then
> > >         extra_paholeopt="${extra_paholeopt} -j"
> > >  fi
> > > +if [ "${pahole_ver}" -ge "124" ]; then
> > > +       # see PAHOLE_HAS_LANG_EXCLUDE
> > > +       extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_enum64"
> > > +fi
> > > 
> > >  echo ${extra_paholeopt}
> > > 
> > > >
> > > > Thanks,
> > > >
> > > >

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Jiri Olsa]

On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > Arnaldo,
> > 
> > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > >
> > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > > >
> > > > >     BTFIDS  vmlinux
> > > > >   + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > >   FAILED: load BTF from vmlinux: Invalid argument
> > > > >
> > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > v1.23 resolves the issue.
> > > > >
> > > > 
> > > > Can you try this, from Martin Reboredo (Archlinux):
> > > > 
> > > > Can you try a build of the kernel or the by passing the
> > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > > 
> > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > > 
> > > This patch helped and kernel builds successfully after applying it.
> > > (Didn't notice this suggestion in release discussion thread.)
> > 
> > Even thought it now compiles with this patch, it does not boot
> > afterwards (in virtme-like env), witch such console messages:
> 
> I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> (since there is no `scripts/pahole-flags.sh`), but I updated
> `scripts/link-vmlinux.sh` with the similar `if` to append
> `--skip_encoding_btf_enum64` which lets then compilation pass.
> 
> Thanks,
> 
> > 
> >   [    0.767649] Run /init as init process
> >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> >   [    0.771262] BPF:size=4 vlen=4
> >   [    0.771511] BPF:
> >   [    0.771680] BPF:Invalid btf_info kind_flag
> >   [    0.772016] BPF:

I can see the same on 5.15, it looks like the libbpf change that
pahole is compiled with is setting the type's kflag for values < 0:
(which is the case for perf_event_task_context enum first value)

  dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API

but IIUC kflag should stay zero for normal enum otherwise the btf meta
verifier screams

if I compile pahole with the libbpf change below I can boot 5.15 kernel
normally

Yonghong, any idea?

thanks,
jirka


---
diff --git a/src/btf.c b/src/btf.c
index 2d14f1a52d7a..53d7516e4b89 100644
--- a/src/btf.c
+++ b/src/btf.c
@@ -2151,10 +2151,6 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value)
 	t = btf_last_type(btf);
 	btf_type_inc_vlen(t);
 
-	/* if negative value, set signedness to signed */
-	if (value < 0)
-		t->info = btf_type_info(btf_kind(t), btf_vlen(t), true);
-
 	btf->hdr->type_len += sz;
 	btf->hdr->str_off += sz;
 	return 0;

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Arnaldo Carvalho de Melo]

Em Fri, Aug 26, 2022 at 03:52:12PM +0200, Jiri Olsa escreveu:
> On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:

> > > > > >     BTFIDS  vmlinux
> > > > > >   + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > > >   FAILED: load BTF from vmlinux: Invalid argument

> > > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > > v1.23 resolves the issue.

> > > > > Can you try this, from Martin Reboredo (Archlinux):

> > > > > Can you try a build of the kernel or the by passing the
> > > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?

> > > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh

> > > > This patch helped and kernel builds successfully after applying it.
> > > > (Didn't notice this suggestion in release discussion thread.)

> > > Even thought it now compiles with this patch, it does not boot
> > > afterwards (in virtme-like env), witch such console messages:

> > I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> > (since there is no `scripts/pahole-flags.sh`), but I updated
> > `scripts/link-vmlinux.sh` with the similar `if` to append
> > `--skip_encoding_btf_enum64` which lets then compilation pass.

> > >   [    0.767649] Run /init as init process
> > >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> > >   [    0.771262] BPF:size=4 vlen=4
> > >   [    0.771511] BPF:
> > >   [    0.771680] BPF:Invalid btf_info kind_flag
> > >   [    0.772016] BPF:
 
> I can see the same on 5.15, it looks like the libbpf change that
> pahole is compiled with is setting the type's kflag for values < 0:
> (which is the case for perf_event_task_context enum first value)
 
>   dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
 
> but IIUC kflag should stay zero for normal enum otherwise the btf meta
> verifier screams
 
> if I compile pahole with the libbpf change below I can boot 5.15 kernel
> normally
 
> Yonghong, any idea?

This made me try to build pahole with the system libbpf instead of with
the one that goes with it, here, testing with libbpf 0.7.0 it wasn't
building as BTF_KIND_ENUM64 came with libbpf 1.0 so I added the
following patch to again allow with the system libbpf, i.e. using:

  $ cmake -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=Off

This will return errors if trying to encode or load enum64 tags, but
disabling it, as done with kernels not supporting BTF_KIND_ENUM64 should
now work, can you please test and report results?

Vitaly I checked and alt:p9 has libbpf 0.2, which is really old, unsure
if it would build there, but alt:sisyphus has 0.8.0, so should work
there, please try.

- Arnaldo

From 2bb968b567011f8a3e47706dc11c2a6ec442352c Mon Sep 17 00:00:00 2001
From: Arnaldo Carvalho de Melo <acme@redhat.com>
Date: Fri, 26 Aug 2022 13:18:26 -0300
Subject: [PATCH 1/1] btf: Fix building with system libbpf

Where we may not have newer things, like BTF_KIND_ENUM64.

So we're now again building with -DLIBBPF_EMBEDDED=Off.

Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 btf_encoder.c | 27 +++++++++++++++++++++++++--
 btf_loader.c  |  7 +++++++
 dutil.h       |  4 ++++
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/btf_encoder.c b/btf_encoder.c
index daa8e3b507d4a856..51d9897fbf1bf41f 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -9,12 +9,12 @@
   Copyright (C) Red Hat Inc
  */
 
+#include <linux/btf.h>
 #include "dwarves.h"
 #include "elf_symtab.h"
 #include "btf_encoder.h"
 #include "gobuffer.h"
 
-#include <linux/btf.h>
 #include <bpf/btf.h>
 #include <bpf/libbpf.h>
 #include <ctype.h> /* for isalpha() and isalnum() */
@@ -124,7 +124,7 @@ static int btf_var_secinfo_cmp(const void *a, const void *b)
 #define BITS_ROUNDDOWN_BYTES(bits) ((bits) >> 3)
 #define BITS_ROUNDUP_BYTES(bits) (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
 
-static const char * const btf_kind_str[NR_BTF_KINDS] = {
+static const char * const btf_kind_str[] = {
 	[BTF_KIND_UNKN]		= "UNKNOWN",
 	[BTF_KIND_INT]		= "INT",
 	[BTF_KIND_PTR]		= "PTR",
@@ -491,6 +491,29 @@ static int32_t btf_encoder__add_struct(struct btf_encoder *encoder, uint8_t kind
 	return id;
 }
 
+#if LIBBPF_MAJOR_VERSION < 1
+static inline int libbpf_err(int ret)
+{
+        if (ret < 0)
+                errno = -ret;
+        return ret;
+}
+
+static
+int btf__add_enum64(struct btf *btf __maybe_unused, const char *name __maybe_unused,
+		    __u32 byte_sz __maybe_unused, bool is_signed __maybe_unused)
+{
+	return  libbpf_err(-ENOTSUP);
+}
+
+static
+int btf__add_enum64_value(struct btf *btf __maybe_unused, const char *name __maybe_unused,
+			  __u64 value __maybe_unused)
+{
+	return  libbpf_err(-ENOTSUP);
+}
+#endif
+
 static int32_t btf_encoder__add_enum(struct btf_encoder *encoder, const char *name, struct type *etype,
 				     struct conf_load *conf_load)
 {
diff --git a/btf_loader.c b/btf_loader.c
index 406a007b61fd4014..69b63a52f591eb84 100644
--- a/btf_loader.c
+++ b/btf_loader.c
@@ -312,6 +312,7 @@ out_free:
 	return -ENOMEM;
 }
 
+#if LIBBPF_MAJOR_VERSION >= 1
 static struct enumerator *enumerator__new64(const char *name, uint64_t value)
 {
 	struct enumerator *en = tag__alloc(sizeof(*en));
@@ -354,6 +355,12 @@ out_free:
 	enumeration__delete(enumeration);
 	return -ENOMEM;
 }
+#else
+static int create_new_enumeration64(struct cu *cu __maybe_unused, const struct btf_type *tp __maybe_unused, uint32_t id __maybe_unused)
+{
+	return -ENOTSUP;
+}
+#endif
 
 static int create_new_subroutine_type(struct cu *cu, const struct btf_type *tp, uint32_t id)
 {
diff --git a/dutil.h b/dutil.h
index e45bba05d05d3725..335a17c07c80e28e 100644
--- a/dutil.h
+++ b/dutil.h
@@ -344,4 +344,8 @@ void __zfree(void **ptr);
 
 #define zfree(ptr) __zfree((void **)(ptr))
 
+#ifndef BTF_KIND_ENUM64
+#define BTF_KIND_ENUM64 19
+#endif
+
 #endif /* _DUTIL_H_ */
-- 
2.37.2

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

Arnaldo,

On Fri, Aug 26, 2022 at 01:37:24PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Aug 26, 2022 at 03:52:12PM +0200, Jiri Olsa escreveu:
> > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> 
> > > >   [    0.767649] Run /init as init process
> > > >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > >   [    0.771262] BPF:size=4 vlen=4
> > > >   [    0.771511] BPF:
> > > >   [    0.771680] BPF:Invalid btf_info kind_flag
> > > >   [    0.772016] BPF:
>  
> > I can see the same on 5.15, it looks like the libbpf change that
> > pahole is compiled with is setting the type's kflag for values < 0:
> > (which is the case for perf_event_task_context enum first value)
>  
> >   dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
>  
> > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > verifier screams
>  
> > if I compile pahole with the libbpf change below I can boot 5.15 kernel
> > normally
>  
> > Yonghong, any idea?
> 
> This made me try to build pahole with the system libbpf instead of with
> the one that goes with it, here, testing with libbpf 0.7.0 it wasn't
> building as BTF_KIND_ENUM64 came with libbpf 1.0 so I added the
> following patch to again allow with the system libbpf, i.e. using:
> 
>   $ cmake -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=Off
> 
> This will return errors if trying to encode or load enum64 tags, but
> disabling it, as done with kernels not supporting BTF_KIND_ENUM64 should
> now work, can you please test and report results?
> 
> Vitaly I checked and alt:p9 has libbpf 0.2, which is really old, unsure
> if it would build there, but alt:sisyphus has 0.8.0, so should work
> there, please try.

Perhaps this does not need to be tested since Jiri found the solution?

BTW, alt:p9 is older ALT branch only for security updates, current
branch is alt:p10, so it's relevant to replace p9 testing with p10, for
new features.

I was building dwarves with -DLIBBPF_EMBEDDED=ON, thinking it would be
more stable (since they updated not independently from each other), is
it recommended to turn in OFF?

ps. I think something needs to be changed somewhere (in kernel src?) or
more users will report build failures when switching to the new pahole.

Thanks,

> 
> - Arnaldo

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Arnaldo Carvalho de Melo]

Em Sat, Aug 27, 2022 at 08:33:10PM +0300, Vitaly Chikunov escreveu:
> On Fri, Aug 26, 2022 at 01:37:24PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Fri, Aug 26, 2022 at 03:52:12PM +0200, Jiri Olsa escreveu:
> > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > >   [    0.767649] Run /init as init process
> > > > >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > >   [    0.771262] BPF:size=4 vlen=4
> > > > >   [    0.771511] BPF:
> > > > >   [    0.771680] BPF:Invalid btf_info kind_flag
> > > > >   [    0.772016] BPF:

> > > I can see the same on 5.15, it looks like the libbpf change that
> > > pahole is compiled with is setting the type's kflag for values < 0:
> > > (which is the case for perf_event_task_context enum first value)

> > >   dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API

> > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > verifier screams

> > > if I compile pahole with the libbpf change below I can boot 5.15 kernel
> > > normally

> > > Yonghong, any idea?

> > This made me try to build pahole with the system libbpf instead of with
> > the one that goes with it, here, testing with libbpf 0.7.0 it wasn't
> > building as BTF_KIND_ENUM64 came with libbpf 1.0 so I added the
> > following patch to again allow with the system libbpf, i.e. using:

> >   $ cmake -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=Off

> > This will return errors if trying to encode or load enum64 tags, but
> > disabling it, as done with kernels not supporting BTF_KIND_ENUM64 should
> > now work, can you please test and report results?

> > Vitaly I checked and alt:p9 has libbpf 0.2, which is really old, unsure
> > if it would build there, but alt:sisyphus has 0.8.0, so should work
> > there, please try.

> Perhaps this does not need to be tested since Jiri found the solution?

Right, what I suggested was just an alternative, to use the system
libbpf package instead of the one that ships with pahole, as the older
libbpf woudln't have the problem hypothesized by Jiri.

But it has already been determined that the problem is elsewhere, so
this just was useful to make me realize building with libbpf-devel was
broken due to BTF_KIND_ENUM64, which should be fixed with this patch.
 
> BTW, alt:p9 is older ALT branch only for security updates, current
> branch is alt:p10, so it's relevant to replace p9 testing with p10, for
> new features.

Sure, I go on adding containers and drop them when it is not possible to
build with them, so far this is what I have for alt:

   9    25.26 alt:p8       : Ok  x86_64-alt-linux-gcc (GCC) 5.3.1 20151207 (ALT p8 5.3.1-alt3.M80P.1)
  10    81.96 alt:p9       : Ok  x86_64-alt-linux-gcc (GCC) 8.4.1 20200305 (ALT p9 8.4.1-alt0.p9.1) , clang version 10.0.0
  11    94.60 alt:p10      : Ok  x86_64-alt-linux-gcc (GCC) 10.3.1 20210703 (ALT Sisyphus 10.3.1-alt2) , clang version 11.0.1
  12    77.53 alt:sisyphus : Ok  x86_64-alt-linux-gcc (GCC) 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt1) , ALT Linux Team clang version 13.0.1

p8's clang is too old, disabled, p9 also has:

ENV NO_BUILD_BPF_SKEL=1

clang also too old to build BPF skels, but still good to build perf
proper.

> I was building dwarves with -DLIBBPF_EMBEDDED=ON, thinking it would be
> more stable (since they updated not independently from each other), is
> it recommended to turn in OFF?

What most distros are doing is to ship a libbpf and they expect and work
to make tools use it, which isn't always possible, like the patch I sent
shows.

Its a balance on having the latest and greatest libbpf that is needed
for some feature pahole uses and the desire to not link anything
statically due to security concerns (having to fix all things linking
statically with a faulty library).
 
> ps. I think something needs to be changed somewhere (in kernel src?) or
> more users will report build failures when switching to the new pahole.

Right, I voiced concerns already about having enum64 encoded by default,
but its not so simple to solve.

People living on the bleeding edge should not have problems, as recent
kernels support the new BTF encodings, its just people building older
kernels with new pahole that will have problems, right?

By test building against alt:p8 you see that I care about such version
mismatches, but its a burden, really, to try and cover all cases.

- Arnaldo

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

On Sat, Aug 27, 2022 at 03:33:53PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Sat, Aug 27, 2022 at 08:33:10PM +0300, Vitaly Chikunov escreveu:
> > On Fri, Aug 26, 2022 at 01:37:24PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Fri, Aug 26, 2022 at 03:52:12PM +0200, Jiri Olsa escreveu:
> > > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > > >   [    0.767649] Run /init as init process
> > > > > >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > > >   [    0.771262] BPF:size=4 vlen=4
> > > > > >   [    0.771511] BPF:
> > > > > >   [    0.771680] BPF:Invalid btf_info kind_flag
> > > > > >   [    0.772016] BPF:
> 
> > > > I can see the same on 5.15, it looks like the libbpf change that
> > > > pahole is compiled with is setting the type's kflag for values < 0:
> > > > (which is the case for perf_event_task_context enum first value)
> 
> > > >   dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> 
> > > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > > verifier screams
> 
> > > > if I compile pahole with the libbpf change below I can boot 5.15 kernel
> > > > normally
> 
> > > > Yonghong, any idea?
> 
> > > This made me try to build pahole with the system libbpf instead of with
> > > the one that goes with it, here, testing with libbpf 0.7.0 it wasn't
> > > building as BTF_KIND_ENUM64 came with libbpf 1.0 so I added the
> > > following patch to again allow with the system libbpf, i.e. using:
> 
> > >   $ cmake -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=Off
> 
> > > This will return errors if trying to encode or load enum64 tags, but
> > > disabling it, as done with kernels not supporting BTF_KIND_ENUM64 should
> > > now work, can you please test and report results?
> 
> > > Vitaly I checked and alt:p9 has libbpf 0.2, which is really old, unsure
> > > if it would build there, but alt:sisyphus has 0.8.0, so should work
> > > there, please try.
> 
> > Perhaps this does not need to be tested since Jiri found the solution?
> 
> Right, what I suggested was just an alternative, to use the system
> libbpf package instead of the one that ships with pahole, as the older
> libbpf woudln't have the problem hypothesized by Jiri.
> 
> But it has already been determined that the problem is elsewhere, so
> this just was useful to make me realize building with libbpf-devel was
> broken due to BTF_KIND_ENUM64, which should be fixed with this patch.
>  
> > BTW, alt:p9 is older ALT branch only for security updates, current
> > branch is alt:p10, so it's relevant to replace p9 testing with p10, for
> > new features.
> 
> Sure, I go on adding containers and drop them when it is not possible to
> build with them, so far this is what I have for alt:
> 
>    9    25.26 alt:p8       : Ok  x86_64-alt-linux-gcc (GCC) 5.3.1 20151207 (ALT p8 5.3.1-alt3.M80P.1)
>   10    81.96 alt:p9       : Ok  x86_64-alt-linux-gcc (GCC) 8.4.1 20200305 (ALT p9 8.4.1-alt0.p9.1) , clang version 10.0.0
>   11    94.60 alt:p10      : Ok  x86_64-alt-linux-gcc (GCC) 10.3.1 20210703 (ALT Sisyphus 10.3.1-alt2) , clang version 11.0.1
>   12    77.53 alt:sisyphus : Ok  x86_64-alt-linux-gcc (GCC) 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt1) , ALT Linux Team clang version 13.0.1

Btw, we also have clang14.0 in sisyphus, it's just non default yet.

> 
> p8's clang is too old, disabled, p9 also has:
> 
> ENV NO_BUILD_BPF_SKEL=1
> 
> clang also too old to build BPF skels, but still good to build perf
> proper.
> 
> > I was building dwarves with -DLIBBPF_EMBEDDED=ON, thinking it would be
> > more stable (since they updated not independently from each other), is
> > it recommended to turn in OFF?
> 
> What most distros are doing is to ship a libbpf and they expect and work
> to make tools use it, which isn't always possible, like the patch I sent
> shows.
> 
> Its a balance on having the latest and greatest libbpf that is needed
> for some feature pahole uses and the desire to not link anything
> statically due to security concerns (having to fix all things linking
> statically with a faulty library).
>  
> > ps. I think something needs to be changed somewhere (in kernel src?) or
> > more users will report build failures when switching to the new pahole.
> 
> Right, I voiced concerns already about having enum64 encoded by default,
> but its not so simple to solve.
> 
> People living on the bleeding edge should not have problems, as recent
> kernels support the new BTF encodings, its just people building older
> kernels with new pahole that will have problems, right?

Yes, but even in Sisyphus (where we try to build bleeding edge stuff) we
build latest stable and longterm kernels, that's why we happen to build
v5.18.19 and v5.15.63 with dwarves 1.24. And 5.15.63 is released just two
days ago so this is not that old.

> By test building against alt:p8 you see that I care about such version
> mismatches, but its a burden, really, to try and cover all cases.

I very much appreciate your work.
Thanks!

> 
> - Arnaldo

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Yonghong Song]

On 8/26/22 6:52 AM, Jiri Olsa wrote:
> On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
>> On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
>>> Arnaldo,
>>>
>>> On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
>>>> On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
>>>>> On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
>>>>>>
>>>>>> I also noticed that after upgrading pahole to v1.24 kernel build (tested on
>>>>>> v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
>>>>>>
>>>>>>      BTFIDS  vmlinux
>>>>>>    + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
>>>>>>    FAILED: load BTF from vmlinux: Invalid argument
>>>>>>
>>>>>> Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
>>>>>> v1.23 resolves the issue.
>>>>>>
>>>>>
>>>>> Can you try this, from Martin Reboredo (Archlinux):
>>>>>
>>>>> Can you try a build of the kernel or the by passing the
>>>>> --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
>>>>>
>>>>> Here's a patch for either in tree scripts/pahole-flags.sh or
>>>>> /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
>>>>
>>>> This patch helped and kernel builds successfully after applying it.
>>>> (Didn't notice this suggestion in release discussion thread.)
>>>
>>> Even thought it now compiles with this patch, it does not boot
>>> afterwards (in virtme-like env), witch such console messages:
>>
>> I'm talking here about 5.15.62. Yes, proposed patch does not apply there
>> (since there is no `scripts/pahole-flags.sh`), but I updated
>> `scripts/link-vmlinux.sh` with the similar `if` to append
>> `--skip_encoding_btf_enum64` which lets then compilation pass.
>>
>> Thanks,
>>
>>>
>>>    [    0.767649] Run /init as init process
>>>    [    0.770858] BPF:[593] ENUM perf_event_task_context
>>>    [    0.771262] BPF:size=4 vlen=4
>>>    [    0.771511] BPF:
>>>    [    0.771680] BPF:Invalid btf_info kind_flag
>>>    [    0.772016] BPF:
> 
> I can see the same on 5.15, it looks like the libbpf change that
> pahole is compiled with is setting the type's kflag for values < 0:
> (which is the case for perf_event_task_context enum first value)
> 
>    dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> 
> but IIUC kflag should stay zero for normal enum otherwise the btf meta
> verifier screams

This is deliberate so we can have sign bit set properly for 32bit enum.
To avoid this behavior, the correct way is to turn off enum64 support
in pahole with flag --skip_encoding_btf_enum64.

> 
> if I compile pahole with the libbpf change below I can boot 5.15 kernel
> normally
> 
> Yonghong, any idea?
> 
> thanks,
> jirka
> 
> 
> ---
> diff --git a/src/btf.c b/src/btf.c
> index 2d14f1a52d7a..53d7516e4b89 100644
> --- a/src/btf.c
> +++ b/src/btf.c
> @@ -2151,10 +2151,6 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value)
>   	t = btf_last_type(btf);
>   	btf_type_inc_vlen(t);
>   
> -	/* if negative value, set signedness to signed */
> -	if (value < 0)
> -		t->info = btf_type_info(btf_kind(t), btf_vlen(t), true);
> -
>   	btf->hdr->type_len += sz;
>   	btf->hdr->str_off += sz;
>   	return 0;

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Jiri Olsa]

On Fri, Aug 26, 2022 at 09:51:54AM -0700, Yonghong Song wrote:
> 
> 
> On 8/26/22 6:52 AM, Jiri Olsa wrote:
> > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > > > Arnaldo,
> > > > 
> > > > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > > > > 
> > > > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > > > > > 
> > > > > > >      BTFIDS  vmlinux
> > > > > > >    + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > > > >    FAILED: load BTF from vmlinux: Invalid argument
> > > > > > > 
> > > > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > > > v1.23 resolves the issue.
> > > > > > > 
> > > > > > 
> > > > > > Can you try this, from Martin Reboredo (Archlinux):
> > > > > > 
> > > > > > Can you try a build of the kernel or the by passing the
> > > > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > > > > 
> > > > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > > > > 
> > > > > This patch helped and kernel builds successfully after applying it.
> > > > > (Didn't notice this suggestion in release discussion thread.)
> > > > 
> > > > Even thought it now compiles with this patch, it does not boot
> > > > afterwards (in virtme-like env), witch such console messages:
> > > 
> > > I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> > > (since there is no `scripts/pahole-flags.sh`), but I updated
> > > `scripts/link-vmlinux.sh` with the similar `if` to append
> > > `--skip_encoding_btf_enum64` which lets then compilation pass.
> > > 
> > > Thanks,
> > > 
> > > > 
> > > >    [    0.767649] Run /init as init process
> > > >    [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > >    [    0.771262] BPF:size=4 vlen=4
> > > >    [    0.771511] BPF:
> > > >    [    0.771680] BPF:Invalid btf_info kind_flag
> > > >    [    0.772016] BPF:
> > 
> > I can see the same on 5.15, it looks like the libbpf change that
> > pahole is compiled with is setting the type's kflag for values < 0:
> > (which is the case for perf_event_task_context enum first value)
> > 
> >    dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> > 
> > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > verifier screams
> 
> This is deliberate so we can have sign bit set properly for 32bit enum.
> To avoid this behavior, the correct way is to turn off enum64 support
> in pahole with flag --skip_encoding_btf_enum64.

I used that as well, it wouldn't compile without

the error is during the boot where the standard enum has kflag set

jirka

> 
> > 
> > if I compile pahole with the libbpf change below I can boot 5.15 kernel
> > normally
> > 
> > Yonghong, any idea?
> > 
> > thanks,
> > jirka
> > 
> > 
> > ---
> > diff --git a/src/btf.c b/src/btf.c
> > index 2d14f1a52d7a..53d7516e4b89 100644
> > --- a/src/btf.c
> > +++ b/src/btf.c
> > @@ -2151,10 +2151,6 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value)
> >   	t = btf_last_type(btf);
> >   	btf_type_inc_vlen(t);
> > -	/* if negative value, set signedness to signed */
> > -	if (value < 0)
> > -		t->info = btf_type_info(btf_kind(t), btf_vlen(t), true);
> > -
> >   	btf->hdr->type_len += sz;
> >   	btf->hdr->str_off += sz;
> >   	return 0;

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Yonghong Song]

On 8/26/22 10:01 AM, Jiri Olsa wrote:
> On Fri, Aug 26, 2022 at 09:51:54AM -0700, Yonghong Song wrote:
>>
>>
>> On 8/26/22 6:52 AM, Jiri Olsa wrote:
>>> On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
>>>> On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
>>>>> Arnaldo,
>>>>>
>>>>> On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
>>>>>> On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
>>>>>>> On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
>>>>>>>>
>>>>>>>> I also noticed that after upgrading pahole to v1.24 kernel build (tested on
>>>>>>>> v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
>>>>>>>>
>>>>>>>>       BTFIDS  vmlinux
>>>>>>>>     + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
>>>>>>>>     FAILED: load BTF from vmlinux: Invalid argument
>>>>>>>>
>>>>>>>> Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
>>>>>>>> v1.23 resolves the issue.
>>>>>>>>
>>>>>>>
>>>>>>> Can you try this, from Martin Reboredo (Archlinux):
>>>>>>>
>>>>>>> Can you try a build of the kernel or the by passing the
>>>>>>> --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
>>>>>>>
>>>>>>> Here's a patch for either in tree scripts/pahole-flags.sh or
>>>>>>> /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
>>>>>>
>>>>>> This patch helped and kernel builds successfully after applying it.
>>>>>> (Didn't notice this suggestion in release discussion thread.)
>>>>>
>>>>> Even thought it now compiles with this patch, it does not boot
>>>>> afterwards (in virtme-like env), witch such console messages:
>>>>
>>>> I'm talking here about 5.15.62. Yes, proposed patch does not apply there
>>>> (since there is no `scripts/pahole-flags.sh`), but I updated
>>>> `scripts/link-vmlinux.sh` with the similar `if` to append
>>>> `--skip_encoding_btf_enum64` which lets then compilation pass.
>>>>
>>>> Thanks,
>>>>
>>>>>
>>>>>     [    0.767649] Run /init as init process
>>>>>     [    0.770858] BPF:[593] ENUM perf_event_task_context
>>>>>     [    0.771262] BPF:size=4 vlen=4
>>>>>     [    0.771511] BPF:
>>>>>     [    0.771680] BPF:Invalid btf_info kind_flag
>>>>>     [    0.772016] BPF:
>>>
>>> I can see the same on 5.15, it looks like the libbpf change that
>>> pahole is compiled with is setting the type's kflag for values < 0:
>>> (which is the case for perf_event_task_context enum first value)
>>>
>>>     dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
>>>
>>> but IIUC kflag should stay zero for normal enum otherwise the btf meta
>>> verifier screams
>>
>> This is deliberate so we can have sign bit set properly for 32bit enum.
>> To avoid this behavior, the correct way is to turn off enum64 support
>> in pahole with flag --skip_encoding_btf_enum64.
> 
> I used that as well, it wouldn't compile without
> 
> the error is during the boot where the standard enum has kflag set

I just tried latest bpf-next, using pahole 1.24, with and without
--skip_encoding_btf_enum64. The following are BTF encoding
for enum perf_event_task_context.

enum perf_event_task_context {
         perf_invalid_context = -1,
         perf_hw_context = 0,
         perf_sw_context,
         perf_nr_task_contexts,
};

With --skip_encoding_btf_enum64:
[2285] ENUM 'perf_event_task_context' encoding=UNSIGNED size=4 vlen=4
         'perf_invalid_context' val=4294967295
         'perf_hw_context' val=0
         'perf_sw_context' val=1
         'perf_nr_task_contexts' val=2

Without --skip_encoding_btf_enum64:
[3786] ENUM 'perf_event_task_context' encoding=SIGNED size=4 vlen=4
         'perf_invalid_context' val=-1
         'perf_hw_context' val=0
         'perf_sw_context' val=1
         'perf_nr_task_contexts' val=2

encoding SIGNED means kflag = 1 and UNSIGNED is the default meaning
kflag = 0. So it looks okay to me. Could you try to use latest
bpftool to dump vmlinux BTF for your vmlinux binary?

Regarding the corresponding pahole enum64 support, we have
the following code,

         if (conf_load->skip_encoding_btf_enum64)
                 err = btf__add_enum_value(encoder->btf, name, 
(uint32_t)value);
         else if (etype->size > 32)
                 err = btf__add_enum64_value(encoder->btf, name, value);
         else
                 err = btf__add_enum_value(encoder->btf, name, value);

If skip_encoding_btf_enum64 is enabled, the value will be passed
as '(uint32_t)value', so '__s64 value' in the parameter should be
a unsigned value and 'if (value < 0) ...' should not be
triggered if skip_encoding_btf_enum64 is enabled.

Jiri, could you double check your build environment?

> 
> jirka
> 
>>
>>>
>>> if I compile pahole with the libbpf change below I can boot 5.15 kernel
>>> normally
>>>
>>> Yonghong, any idea?
>>>
>>> thanks,
>>> jirka
>>>
>>>
>>> ---
>>> diff --git a/src/btf.c b/src/btf.c
>>> index 2d14f1a52d7a..53d7516e4b89 100644
>>> --- a/src/btf.c
>>> +++ b/src/btf.c
>>> @@ -2151,10 +2151,6 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value)
>>>    	t = btf_last_type(btf);
>>>    	btf_type_inc_vlen(t);
>>> -	/* if negative value, set signedness to signed */
>>> -	if (value < 0)
>>> -		t->info = btf_type_info(btf_kind(t), btf_vlen(t), true);
>>> -
>>>    	btf->hdr->type_len += sz;
>>>    	btf->hdr->str_off += sz;
>>>    	return 0;

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Jiri Olsa]

On Fri, Aug 26, 2022 at 11:53:34AM -0700, Yonghong Song wrote:
> 
> 
> On 8/26/22 10:01 AM, Jiri Olsa wrote:
> > On Fri, Aug 26, 2022 at 09:51:54AM -0700, Yonghong Song wrote:
> > > 
> > > 
> > > On 8/26/22 6:52 AM, Jiri Olsa wrote:
> > > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > > On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > > > > > Arnaldo,
> > > > > > 
> > > > > > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > > > > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > > > > > > 
> > > > > > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > > > > > > > 
> > > > > > > > >       BTFIDS  vmlinux
> > > > > > > > >     + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > > > > > >     FAILED: load BTF from vmlinux: Invalid argument
> > > > > > > > > 
> > > > > > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > > > > > v1.23 resolves the issue.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Can you try this, from Martin Reboredo (Archlinux):
> > > > > > > > 
> > > > > > > > Can you try a build of the kernel or the by passing the
> > > > > > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > > > > > > 
> > > > > > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > > > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > > > > > > 
> > > > > > > This patch helped and kernel builds successfully after applying it.
> > > > > > > (Didn't notice this suggestion in release discussion thread.)
> > > > > > 
> > > > > > Even thought it now compiles with this patch, it does not boot
> > > > > > afterwards (in virtme-like env), witch such console messages:
> > > > > 
> > > > > I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> > > > > (since there is no `scripts/pahole-flags.sh`), but I updated
> > > > > `scripts/link-vmlinux.sh` with the similar `if` to append
> > > > > `--skip_encoding_btf_enum64` which lets then compilation pass.
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > > 
> > > > > >     [    0.767649] Run /init as init process
> > > > > >     [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > > >     [    0.771262] BPF:size=4 vlen=4
> > > > > >     [    0.771511] BPF:
> > > > > >     [    0.771680] BPF:Invalid btf_info kind_flag
> > > > > >     [    0.772016] BPF:
> > > > 
> > > > I can see the same on 5.15, it looks like the libbpf change that
> > > > pahole is compiled with is setting the type's kflag for values < 0:
> > > > (which is the case for perf_event_task_context enum first value)
> > > > 
> > > >     dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> > > > 
> > > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > > verifier screams
> > > 
> > > This is deliberate so we can have sign bit set properly for 32bit enum.
> > > To avoid this behavior, the correct way is to turn off enum64 support
> > > in pahole with flag --skip_encoding_btf_enum64.
> > 
> > I used that as well, it wouldn't compile without
> > 
> > the error is during the boot where the standard enum has kflag set
> 
> I just tried latest bpf-next, using pahole 1.24, with and without
> --skip_encoding_btf_enum64. The following are BTF encoding
> for enum perf_event_task_context.
> 
> enum perf_event_task_context {
>         perf_invalid_context = -1,
>         perf_hw_context = 0,
>         perf_sw_context,
>         perf_nr_task_contexts,
> };
> 
> With --skip_encoding_btf_enum64:
> [2285] ENUM 'perf_event_task_context' encoding=UNSIGNED size=4 vlen=4
>         'perf_invalid_context' val=4294967295
>         'perf_hw_context' val=0
>         'perf_sw_context' val=1
>         'perf_nr_task_contexts' val=2
> 
> Without --skip_encoding_btf_enum64:
> [3786] ENUM 'perf_event_task_context' encoding=SIGNED size=4 vlen=4
>         'perf_invalid_context' val=-1
>         'perf_hw_context' val=0
>         'perf_sw_context' val=1
>         'perf_nr_task_contexts' val=2
> 
> encoding SIGNED means kflag = 1 and UNSIGNED is the default meaning
> kflag = 0. So it looks okay to me. Could you try to use latest
> bpftool to dump vmlinux BTF for your vmlinux binary?
> 
> Regarding the corresponding pahole enum64 support, we have
> the following code,
> 
>         if (conf_load->skip_encoding_btf_enum64)
>                 err = btf__add_enum_value(encoder->btf, name,
> (uint32_t)value);
>         else if (etype->size > 32)
>                 err = btf__add_enum64_value(encoder->btf, name, value);
>         else
>                 err = btf__add_enum_value(encoder->btf, name, value);
> 
> If skip_encoding_btf_enum64 is enabled, the value will be passed
> as '(uint32_t)value', so '__s64 value' in the parameter should be
> a unsigned value and 'if (value < 0) ...' should not be
> triggered if skip_encoding_btf_enum64 is enabled.
> 
> Jiri, could you double check your build environment?

ah right.. it's the build problem, 5.15 is missing backport of
  9741e07ece7c kbuild: Unify options for BTF generation for vmlinux and modules

so modules BTF build does not pick up the --skip_encoding_btf_enum64,
with the 5.15.61 extra change below I can boot the kernel properly

sorry for the noise

Vitaly, could you please try with the change below?

thanks,
jirka


---
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index ff805777431c..47717e09000b 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -40,7 +40,7 @@ quiet_cmd_ld_ko_o = LD [M]  $@
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
 	if [ -f vmlinux ]; then						\
-		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --btf_base vmlinux $@; \
+		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --skip_encoding_btf_enum64 --btf_base vmlinux $@; \
 	else								\
 		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
 	fi;

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

Jiri,

On Fri, Aug 26, 2022 at 10:19:55PM +0200, Jiri Olsa wrote:
> On Fri, Aug 26, 2022 at 11:53:34AM -0700, Yonghong Song wrote:
> > On 8/26/22 10:01 AM, Jiri Olsa wrote:
> > > On Fri, Aug 26, 2022 at 09:51:54AM -0700, Yonghong Song wrote:
> > > > On 8/26/22 6:52 AM, Jiri Olsa wrote:
> > > > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > > > On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > > > > > > Arnaldo,
> > > > > > > 
> > > > > > > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > > > > > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > > > > > > > 
> > > > > > > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > > > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > > > > > > > > 
> > > > > > > > > >       BTFIDS  vmlinux
> > > > > > > > > >     + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > > > > > > >     FAILED: load BTF from vmlinux: Invalid argument
> > > > > > > > > > 
> > > > > > > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > > > > > > v1.23 resolves the issue.
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Can you try this, from Martin Reboredo (Archlinux):
> > > > > > > > > 
> > > > > > > > > Can you try a build of the kernel or the by passing the
> > > > > > > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > > > > > > > 
> > > > > > > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > > > > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > > > > > > > 
> > > > > > > > This patch helped and kernel builds successfully after applying it.
> > > > > > > > (Didn't notice this suggestion in release discussion thread.)
> > > > > > > 
> > > > > > > Even thought it now compiles with this patch, it does not boot
> > > > > > > afterwards (in virtme-like env), witch such console messages:
> > > > > > 
> > > > > > I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> > > > > > (since there is no `scripts/pahole-flags.sh`), but I updated
> > > > > > `scripts/link-vmlinux.sh` with the similar `if` to append
> > > > > > `--skip_encoding_btf_enum64` which lets then compilation pass.
> > > > > > 
> > > > > > Thanks,
> > > > > > 
> > > > > > > 
> > > > > > >     [    0.767649] Run /init as init process
> > > > > > >     [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > > > >     [    0.771262] BPF:size=4 vlen=4
> > > > > > >     [    0.771511] BPF:
> > > > > > >     [    0.771680] BPF:Invalid btf_info kind_flag
> > > > > > >     [    0.772016] BPF:
> > > > > 
> > > > > I can see the same on 5.15, it looks like the libbpf change that
> > > > > pahole is compiled with is setting the type's kflag for values < 0:
> > > > > (which is the case for perf_event_task_context enum first value)
> > > > > 
> > > > >     dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> > > > > 
> > > > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > > > verifier screams
> > > > 
> > > > This is deliberate so we can have sign bit set properly for 32bit enum.
> > > > To avoid this behavior, the correct way is to turn off enum64 support
> > > > in pahole with flag --skip_encoding_btf_enum64.
> > > 
> > > I used that as well, it wouldn't compile without
> > > 
> > > the error is during the boot where the standard enum has kflag set
> > 
> > I just tried latest bpf-next, using pahole 1.24, with and without
> > --skip_encoding_btf_enum64. The following are BTF encoding
> > for enum perf_event_task_context.
> > 
> > enum perf_event_task_context {
> >         perf_invalid_context = -1,
> >         perf_hw_context = 0,
> >         perf_sw_context,
> >         perf_nr_task_contexts,
> > };
> > 
> > With --skip_encoding_btf_enum64:
> > [2285] ENUM 'perf_event_task_context' encoding=UNSIGNED size=4 vlen=4
> >         'perf_invalid_context' val=4294967295
> >         'perf_hw_context' val=0
> >         'perf_sw_context' val=1
> >         'perf_nr_task_contexts' val=2
> > 
> > Without --skip_encoding_btf_enum64:
> > [3786] ENUM 'perf_event_task_context' encoding=SIGNED size=4 vlen=4
> >         'perf_invalid_context' val=-1
> >         'perf_hw_context' val=0
> >         'perf_sw_context' val=1
> >         'perf_nr_task_contexts' val=2
> > 
> > encoding SIGNED means kflag = 1 and UNSIGNED is the default meaning
> > kflag = 0. So it looks okay to me. Could you try to use latest
> > bpftool to dump vmlinux BTF for your vmlinux binary?
> > 
> > Regarding the corresponding pahole enum64 support, we have
> > the following code,
> > 
> >         if (conf_load->skip_encoding_btf_enum64)
> >                 err = btf__add_enum_value(encoder->btf, name,
> > (uint32_t)value);
> >         else if (etype->size > 32)
> >                 err = btf__add_enum64_value(encoder->btf, name, value);
> >         else
> >                 err = btf__add_enum_value(encoder->btf, name, value);
> > 
> > If skip_encoding_btf_enum64 is enabled, the value will be passed
> > as '(uint32_t)value', so '__s64 value' in the parameter should be
> > a unsigned value and 'if (value < 0) ...' should not be
> > triggered if skip_encoding_btf_enum64 is enabled.
> > 
> > Jiri, could you double check your build environment?
> 
> ah right.. it's the build problem, 5.15 is missing backport of
>   9741e07ece7c kbuild: Unify options for BTF generation for vmlinux and modules
> 
> so modules BTF build does not pick up the --skip_encoding_btf_enum64,
> with the 5.15.61 extra change below I can boot the kernel properly
> 
> sorry for the noise
> 
> Vitaly, could you please try with the change below?

I just tested and after this additional change v5.15.63 kernel
started to boot successfully.

Thanks!

> 
> thanks,
> jirka
> 
> 
> ---
> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> index ff805777431c..47717e09000b 100644
> --- a/scripts/Makefile.modfinal
> +++ b/scripts/Makefile.modfinal
> @@ -40,7 +40,7 @@ quiet_cmd_ld_ko_o = LD [M]  $@
>  quiet_cmd_btf_ko = BTF [M] $@
>        cmd_btf_ko = 							\
>  	if [ -f vmlinux ]; then						\
> -		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --btf_base vmlinux $@; \
> +		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --skip_encoding_btf_enum64 --btf_base vmlinux $@; \
>  	else								\
>  		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
>  	fi;

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Yonghong Song]

On 8/25/22 7:59 PM, Vitaly Chikunov wrote:
> On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
>> Arnaldo,
>>
>> On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
>>> On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
>>>> On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
>>>>>
>>>>> I also noticed that after upgrading pahole to v1.24 kernel build (tested on
>>>>> v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
>>>>>
>>>>>      BTFIDS  vmlinux
>>>>>    + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
>>>>>    FAILED: load BTF from vmlinux: Invalid argument
>>>>>
>>>>> Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
>>>>> v1.23 resolves the issue.
>>>>>
>>>>
>>>> Can you try this, from Martin Reboredo (Archlinux):
>>>>
>>>> Can you try a build of the kernel or the by passing the
>>>> --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
>>>>
>>>> Here's a patch for either in tree scripts/pahole-flags.sh or
>>>> /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
>>>
>>> This patch helped and kernel builds successfully after applying it.
>>> (Didn't notice this suggestion in release discussion thread.)
>>
>> Even thought it now compiles with this patch, it does not boot
>> afterwards (in virtme-like env), witch such console messages:
> 
> I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> (since there is no `scripts/pahole-flags.sh`), but I updated
> `scripts/link-vmlinux.sh` with the similar `if` to append
> `--skip_encoding_btf_enum64` which lets then compilation pass.

Right, pahole v1.24 supports enum64 to correctly encode
some big 64bit values in BTF. But enum64 is only supported
in recent kernels. For old kernels, --skip_encoding_btf_enum64
is the way to workaround the issue.

> 
> Thanks,
> 
>>
>>    [    0.767649] Run /init as init process
>>    [    0.770858] BPF:[593] ENUM perf_event_task_context
>>    [    0.771262] BPF:size=4 vlen=4
>>    [    0.771511] BPF:
>>    [    0.771680] BPF:Invalid btf_info kind_flag
>>    [    0.772016] BPF:
>>    [    0.772016]
>>    [    0.772288] failed to validate module [9pnet] BTF: -22
>>    init_module '9pnet.ko' error -1
>>    [    0.785515] 9p: Unknown symbol p9_client_getattr_dotl (err -2)
>>    [    0.786005] 9p: Unknown symbol p9_client_wstat (err -2)
>>    [    0.786438] 9p: Unknown symbol p9_client_open (err -2)
>>    [    0.786863] 9p: Unknown symbol p9_client_rename (err -2)
>>    [    0.787307] 9p: Unknown symbol p9_client_remove (err -2)
>>    [    0.787749] 9p: Unknown symbol p9_client_renameat (err -2)
[...]

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Arnaldo Carvalho de Melo]

Em Fri, Aug 26, 2022 at 09:47:59AM -0700, Yonghong Song escreveu:
> 
> 
> On 8/25/22 7:59 PM, Vitaly Chikunov wrote:
> > On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > > Arnaldo,
> > > 
> > > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > > > 
> > > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > > > > 
> > > > > >      BTFIDS  vmlinux
> > > > > >    + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > > >    FAILED: load BTF from vmlinux: Invalid argument
> > > > > > 
> > > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > > v1.23 resolves the issue.
> > > > > > 
> > > > > 
> > > > > Can you try this, from Martin Reboredo (Archlinux):
> > > > > 
> > > > > Can you try a build of the kernel or the by passing the
> > > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > > > 
> > > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > > > 
> > > > This patch helped and kernel builds successfully after applying it.
> > > > (Didn't notice this suggestion in release discussion thread.)
> > > 
> > > Even thought it now compiles with this patch, it does not boot
> > > afterwards (in virtme-like env), witch such console messages:
> > 
> > I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> > (since there is no `scripts/pahole-flags.sh`), but I updated
> > `scripts/link-vmlinux.sh` with the similar `if` to append
> > `--skip_encoding_btf_enum64` which lets then compilation pass.
> 
> Right, pahole v1.24 supports enum64 to correctly encode
> some big 64bit values in BTF. But enum64 is only supported
> in recent kernels. For old kernels, --skip_encoding_btf_enum64
> is the way to workaround the issue.

Check Jiri response, this doesn't seem to be enough.

- Arnaldo

> > Thanks,
> > 
> > > 
> > >    [    0.767649] Run /init as init process
> > >    [    0.770858] BPF:[593] ENUM perf_event_task_context
> > >    [    0.771262] BPF:size=4 vlen=4
> > >    [    0.771511] BPF:
> > >    [    0.771680] BPF:Invalid btf_info kind_flag
> > >    [    0.772016] BPF:
> > >    [    0.772016]
> > >    [    0.772288] failed to validate module [9pnet] BTF: -22
> > >    init_module '9pnet.ko' error -1
> > >    [    0.785515] 9p: Unknown symbol p9_client_getattr_dotl (err -2)
> > >    [    0.786005] 9p: Unknown symbol p9_client_wstat (err -2)
> > >    [    0.786438] 9p: Unknown symbol p9_client_open (err -2)
> > >    [    0.786863] 9p: Unknown symbol p9_client_rename (err -2)
> > >    [    0.787307] 9p: Unknown symbol p9_client_remove (err -2)
> > >    [    0.787749] 9p: Unknown symbol p9_client_renameat (err -2)
> [...]

-- 

- Arnaldo