Re: Problem with endianess of pahole BTF output for vmlinux

Re: Problem with endianess of pahole BTF output for vmlinux

[Tony Ambardar]

On Wed, 9 Sep 2020 at 07:27, Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
>
> Em Wed, Sep 09, 2020 at 11:02:24AM +0200, Ilya Leoshkevich escreveu:
> > On Tue, 2020-09-08 at 13:18 -0700, Andrii Nakryiko wrote:
> > > On Mon, Sep 7, 2020 at 9:02 AM Ilya Leoshkevich <iii@linux.ibm.com> wrote:
> > > > On Sat, 2020-09-05 at 21:16 -0700, Tony Ambardar wrote:
>
[...]
> > > > > Is this expected? Is DEBUG_INFO_BTF supported in general when
> > > > > cross-compiling? How does one generate BTF encoded for the
> > > > > target endianness with pahole?
>
> The BTF loader has support for endianness, its just the encoder that
> doesn't :-\
>
> I.e. pahole can grok a big endian BTF payload on a little endian machine
> and vice-versa, just can't cross-build BTF payloads ATM.
>
> > > Yes, it's expected, unfortunately. Right now cross-compiling to a
> > > different endianness isn't supported. You can cross-compile only if
> > > target endianness matches host endianness.
>
> I agree that having this in libbpf is better, it should be done as part
> of producing the result of the deduplication phase.
>
Thanks for confirming this wasn't a case of operator error. My platforms for
learning/experimenting with BPF have been small embedded ones, including
cross-compiling to different archs, word-size and endianness, which have
"helped" me run into multiple problems till now. This one is the first I
couldn't work around however...

[...]
> > > I'm working on extending BTF APIs in libbpf at the moment. Switching
> > > endianness would be rather easy once all that is done. With these new
> > > APIs it will be possible to switch pahole to use libbpf APIs to
> > > produce BTF output and support arbitrary endianness as well. Right
> > > now, I'd rather avoid implementing this in pahole, libbpf is a much
> > > better place for this (and will require ongoing updates if/when we
> > > introduce new types and fields to BTF).
>
> Right, we could do it right after btf_dedup() and before
> btf_elf__write(), doing the same process as in btf_loader.c, i.e.
> checking if the ELF target arch is different in endianness and doing the
> reverse of the loader.
>
> > > Hope this plan works for you guys.
> >
> > That sounds really good to me, thanks!
>
Andrii and Arnaldo, I really appreciate your working on a proper endianness fix.
If you have a WIP or staging branch and could use some help please let me know.

Best regards,
Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Andrii Nakryiko]

On Sat, Sep 19, 2020 at 12:58 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> On Wed, 9 Sep 2020 at 07:27, Arnaldo Carvalho de Melo
> <arnaldo.melo@gmail.com> wrote:
> >
> > Em Wed, Sep 09, 2020 at 11:02:24AM +0200, Ilya Leoshkevich escreveu:
> > > On Tue, 2020-09-08 at 13:18 -0700, Andrii Nakryiko wrote:
> > > > On Mon, Sep 7, 2020 at 9:02 AM Ilya Leoshkevich <iii@linux.ibm.com> wrote:
> > > > > On Sat, 2020-09-05 at 21:16 -0700, Tony Ambardar wrote:
> >
> [...]
> > > > > > Is this expected? Is DEBUG_INFO_BTF supported in general when
> > > > > > cross-compiling? How does one generate BTF encoded for the
> > > > > > target endianness with pahole?
> >
> > The BTF loader has support for endianness, its just the encoder that
> > doesn't :-\
> >
> > I.e. pahole can grok a big endian BTF payload on a little endian machine
> > and vice-versa, just can't cross-build BTF payloads ATM.
> >
> > > > Yes, it's expected, unfortunately. Right now cross-compiling to a
> > > > different endianness isn't supported. You can cross-compile only if
> > > > target endianness matches host endianness.
> >
> > I agree that having this in libbpf is better, it should be done as part
> > of producing the result of the deduplication phase.
> >
> Thanks for confirming this wasn't a case of operator error. My platforms for
> learning/experimenting with BPF have been small embedded ones, including
> cross-compiling to different archs, word-size and endianness, which have
> "helped" me run into multiple problems till now. This one is the first I
> couldn't work around however...
>
> [...]
> > > > I'm working on extending BTF APIs in libbpf at the moment. Switching
> > > > endianness would be rather easy once all that is done. With these new
> > > > APIs it will be possible to switch pahole to use libbpf APIs to
> > > > produce BTF output and support arbitrary endianness as well. Right
> > > > now, I'd rather avoid implementing this in pahole, libbpf is a much
> > > > better place for this (and will require ongoing updates if/when we
> > > > introduce new types and fields to BTF).
> >
> > Right, we could do it right after btf_dedup() and before
> > btf_elf__write(), doing the same process as in btf_loader.c, i.e.
> > checking if the ELF target arch is different in endianness and doing the
> > reverse of the loader.
> >
> > > > Hope this plan works for you guys.
> > >
> > > That sounds really good to me, thanks!
> >
> Andrii and Arnaldo, I really appreciate your working on a proper endianness fix.
> If you have a WIP or staging branch and could use some help please let me know.
>

I have a bunch of code changes locally. I'll clean that up, partition
libbpf and pahole patches, and post them for review this week. To
address endianness support, those are the prerequisites. Once those
changes land, I'll be able to solve endianness issues you are having.
So just a bit longer till all that is done, sorry for the wait!

> Best regards,
> Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Andrii Nakryiko]

On Mon, Sep 21, 2020 at 11:19 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sat, Sep 19, 2020 at 12:58 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > On Wed, 9 Sep 2020 at 07:27, Arnaldo Carvalho de Melo
> > <arnaldo.melo@gmail.com> wrote:
> > >
> > > Em Wed, Sep 09, 2020 at 11:02:24AM +0200, Ilya Leoshkevich escreveu:
> > > > On Tue, 2020-09-08 at 13:18 -0700, Andrii Nakryiko wrote:
> > > > > On Mon, Sep 7, 2020 at 9:02 AM Ilya Leoshkevich <iii@linux.ibm.com> wrote:
> > > > > > On Sat, 2020-09-05 at 21:16 -0700, Tony Ambardar wrote:
> > >
> > [...]
> > > > > > > Is this expected? Is DEBUG_INFO_BTF supported in general when
> > > > > > > cross-compiling? How does one generate BTF encoded for the
> > > > > > > target endianness with pahole?
> > >
> > > The BTF loader has support for endianness, its just the encoder that
> > > doesn't :-\
> > >
> > > I.e. pahole can grok a big endian BTF payload on a little endian machine
> > > and vice-versa, just can't cross-build BTF payloads ATM.
> > >
> > > > > Yes, it's expected, unfortunately. Right now cross-compiling to a
> > > > > different endianness isn't supported. You can cross-compile only if
> > > > > target endianness matches host endianness.
> > >
> > > I agree that having this in libbpf is better, it should be done as part
> > > of producing the result of the deduplication phase.
> > >
> > Thanks for confirming this wasn't a case of operator error. My platforms for
> > learning/experimenting with BPF have been small embedded ones, including
> > cross-compiling to different archs, word-size and endianness, which have
> > "helped" me run into multiple problems till now. This one is the first I
> > couldn't work around however...
> >
> > [...]
> > > > > I'm working on extending BTF APIs in libbpf at the moment. Switching
> > > > > endianness would be rather easy once all that is done. With these new
> > > > > APIs it will be possible to switch pahole to use libbpf APIs to
> > > > > produce BTF output and support arbitrary endianness as well. Right
> > > > > now, I'd rather avoid implementing this in pahole, libbpf is a much
> > > > > better place for this (and will require ongoing updates if/when we
> > > > > introduce new types and fields to BTF).
> > >
> > > Right, we could do it right after btf_dedup() and before
> > > btf_elf__write(), doing the same process as in btf_loader.c, i.e.
> > > checking if the ELF target arch is different in endianness and doing the
> > > reverse of the loader.
> > >
> > > > > Hope this plan works for you guys.
> > > >
> > > > That sounds really good to me, thanks!
> > >
> > Andrii and Arnaldo, I really appreciate your working on a proper endianness fix.
> > If you have a WIP or staging branch and could use some help please let me know.
> >
>
> I have a bunch of code changes locally. I'll clean that up, partition
> libbpf and pahole patches, and post them for review this week. To
> address endianness support, those are the prerequisites. Once those
> changes land, I'll be able to solve endianness issues you are having.
> So just a bit longer till all that is done, sorry for the wait!
>

Question to folks that are working with 32-bit and/or big-endian
architectures. Do you guys have an VM image that you'd be able to
share with me, such that I can use it with qemu to test patches like
this. My normal setup is all 64-bit/little-endian, so testing changes
like this (and a few more I'm planning to do to address mixed 32-bit
on the host vs 64-bit in BPF cases) is a bit problematic. And it's
hard to get superpumped about spending lots of time setting up a new
Linux image (never goes easy or fast for me).

So, if you do have something like this, please share. Thank you!

> > Best regards,
> > Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Luka Perkov]

Hello Andrii,

On Mon, Sep 28, 2020 at 10:19 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
> Question to folks that are working with 32-bit and/or big-endian
> architectures. Do you guys have an VM image that you'd be able to
> share with me, such that I can use it with qemu to test patches like
> this. My normal setup is all 64-bit/little-endian, so testing changes
> like this (and a few more I'm planning to do to address mixed 32-bit
> on the host vs 64-bit in BPF cases) is a bit problematic. And it's
> hard to get superpumped about spending lots of time setting up a new
> Linux image (never goes easy or fast for me).
>
> So, if you do have something like this, please share. Thank you!

We thought this would be an issue and we have been working on this but
didn't finalize it yet.

We'll speed it up and send you an easy way to replicate the
development environment.

Luka and Juraj will also share some of our additional findings
regarding 32bit ARM.

Thanks,
Luka

Re: Problem with endianess of pahole BTF output for vmlinux

[Tony Ambardar]

Hello Andrii!

On Mon, 28 Sep 2020 at 13:19, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> On Mon, Sep 21, 2020 at 11:19 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > I have a bunch of code changes locally. I'll clean that up, partition
> > libbpf and pahole patches, and post them for review this week. To
> > address endianness support, those are the prerequisites. Once those
> > changes land, I'll be able to solve endianness issues you are having.
> > So just a bit longer till all that is done, sorry for the wait!
> >
>
> Question to folks that are working with 32-bit and/or big-endian
> architectures. Do you guys have an VM image that you'd be able to
> share with me, such that I can use it with qemu to test patches like
> this. My normal setup is all 64-bit/little-endian, so testing changes
> like this (and a few more I'm planning to do to address mixed 32-bit
> on the host vs 64-bit in BPF cases) is a bit problematic. And it's
> hard to get superpumped about spending lots of time setting up a new
> Linux image (never goes easy or fast for me).
>
> So, if you do have something like this, please share. Thank you!
>

I can provide 32-bit and 64-bit big-endian system images for running
on QEMU's malta target. These are built using OpenWRT's build system
and include a recent stable bpftool (v5.8.x) and v5.4.x kernel. Is
that sufficient? It would work if manually creating raw or elf-based
BTF files on a build host, then copying into the QEMU target to test
parsing with bpftool (linked with the standard libbpf).

For changes to the Linux build system itself (e.g. pahole endian
options and target endian awareness), you would need to set up a
standard OpenWRT build environment. I can help with that, or simply
integrate your patches myself for testing. As you say, nothing to be
super pumped about...

Let me know what's easiest and how best to get images to you.

Many thanks,
Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Andrii Nakryiko]

On Mon, Sep 28, 2020 at 8:41 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> Hello Andrii!
>
> On Mon, 28 Sep 2020 at 13:19, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >
> > On Mon, Sep 21, 2020 at 11:19 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > I have a bunch of code changes locally. I'll clean that up, partition
> > > libbpf and pahole patches, and post them for review this week. To
> > > address endianness support, those are the prerequisites. Once those
> > > changes land, I'll be able to solve endianness issues you are having.
> > > So just a bit longer till all that is done, sorry for the wait!
> > >
> >
> > Question to folks that are working with 32-bit and/or big-endian
> > architectures. Do you guys have an VM image that you'd be able to
> > share with me, such that I can use it with qemu to test patches like
> > this. My normal setup is all 64-bit/little-endian, so testing changes
> > like this (and a few more I'm planning to do to address mixed 32-bit
> > on the host vs 64-bit in BPF cases) is a bit problematic. And it's
> > hard to get superpumped about spending lots of time setting up a new
> > Linux image (never goes easy or fast for me).
> >
> > So, if you do have something like this, please share. Thank you!
> >
>
> I can provide 32-bit and 64-bit big-endian system images for running
> on QEMU's malta target. These are built using OpenWRT's build system
> and include a recent stable bpftool (v5.8.x) and v5.4.x kernel. Is
> that sufficient? It would work if manually creating raw or elf-based
> BTF files on a build host, then copying into the QEMU target to test
> parsing with bpftool (linked with the standard libbpf).

That would be great! I intend to run them under qemu-system-arm and
supply latest kernel through -kernel option, so kernel itself is not
that critical. Same for bpftool, pahole, etc, I'll just supply them
from my host environment. So please let me know how I can get ahold of
those. Sample qemu invocation command line would be highly appreciated
as well. Thank you!

>
> For changes to the Linux build system itself (e.g. pahole endian
> options and target endian awareness), you would need to set up a

I think that shouldn't be a problem and should be handled
transparently, even in a cross-compilation case, but let's see.

> standard OpenWRT build environment. I can help with that, or simply
> integrate your patches myself for testing. As you say, nothing to be
> super pumped about...
>
> Let me know what's easiest and how best to get images to you.

Any way you like and can. Dropbox, Google drive, what have you.

>
> Many thanks,
> Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Tony Ambardar]

On Mon, 28 Sep 2020 at 21:15, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> On Mon, Sep 28, 2020 at 8:41 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
[...]
> > I can provide 32-bit and 64-bit big-endian system images for running
> > on QEMU's malta target. These are built using OpenWRT's build system
> > and include a recent stable bpftool (v5.8.x) and v5.4.x kernel. Is
> > that sufficient? It would work if manually creating raw or elf-based
> > BTF files on a build host, then copying into the QEMU target to test
> > parsing with bpftool (linked with the standard libbpf).
>
> That would be great! I intend to run them under qemu-system-arm and
> supply latest kernel through -kernel option, so kernel itself is not
> that critical. Same for bpftool, pahole, etc, I'll just supply them
> from my host environment. So please let me know how I can get ahold of
> those. Sample qemu invocation command line would be highly appreciated
> as well. Thank you!
>
Sounds good. However, malta is actually a MIPS platform. I've been using it
a long time because it makes things particularly easy to switch configuration
between different word-sizes and endianness.

I had some malta mips images ready to go, but if you need ARM I'll need
to look into building images for big-endian ARM. Big-endian isn't so common
in the wild, and I'll need to see if OpenWRT supports these, and how to
configure with QEMU's 'armvirt' target if possible...

> >
> > For changes to the Linux build system itself (e.g. pahole endian
> > options and target endian awareness), you would need to set up a
>
> I think that shouldn't be a problem and should be handled
> transparently, even in a cross-compilation case, but let's see.
>
> > standard OpenWRT build environment. I can help with that, or simply
> > integrate your patches myself for testing. As you say, nothing to be
> > super pumped about...
> >
> > Let me know what's easiest and how best to get images to you.
>
> Any way you like and can. Dropbox, Google drive, what have you.
>
Meantime, I can package up what I have and send you the details. That
would include images for mips32/64 big-endian and arm32/64 little-endian,
plus usage examples. Is that still helpful for you?

Thanks,
Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Juraj Vijtiuk]

Hello Andrii and Tony,

as Luka mentioned we do not yet have a final version of the testing
environment that we are working on for ARM32.

However, I previously created an ARM32 little endian QEMU virt image
that runs Debian, to test issues that we encountered with BPF CO-RE
programs on ARM32.
I only managed to get BPF CO-RE programs working with the default
Debian kernel image, which is version 4.19. I built a custom 5.7
kernel but it has some issues (none of the BPF programs work), which i
haven't managed to fix yet.
I'm not sure if the issue is due to the 5.7 kernel image, or due to
the QEMU image. Because of that I'm not sure if it will work with
other kernel versions, but if you are interested in trying to use it,
it is available at the following Google Drive link:
https://drive.google.com/drive/folders/1hs80tYhQG76As7_vkj1y7EF-sFXWxep1?usp=sharing

To run the command i use "qemu-system-arm -M virt -m 1024 -kernel
vmlinuz-4.19.0-10-armmp-lpae -initrd initrd.img-4.19.0-10-armmp-lpae
-append 'root=/dev/vda2' -drive
if=none,file=qemu-backup.qcow2,format=qcow2,id=hd -device
virtio-blk-device,drive=hd -netdev user,id=mynet -device
virtio-net-device,netdev=mynet -nographic".

The logins are root:test and user:user.
The link has the disk image in qemu-backup.qcow, the 4.19 kernel and
initrd and a run.sh which has the above mentioned command to run the
image.

Regards,
Juraj


On Tue, Sep 29, 2020 at 8:48 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> On Mon, 28 Sep 2020 at 21:15, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >
> > On Mon, Sep 28, 2020 at 8:41 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > >
> [...]
> > > I can provide 32-bit and 64-bit big-endian system images for running
> > > on QEMU's malta target. These are built using OpenWRT's build system
> > > and include a recent stable bpftool (v5.8.x) and v5.4.x kernel. Is
> > > that sufficient? It would work if manually creating raw or elf-based
> > > BTF files on a build host, then copying into the QEMU target to test
> > > parsing with bpftool (linked with the standard libbpf).
> >
> > That would be great! I intend to run them under qemu-system-arm and
> > supply latest kernel through -kernel option, so kernel itself is not
> > that critical. Same for bpftool, pahole, etc, I'll just supply them
> > from my host environment. So please let me know how I can get ahold of
> > those. Sample qemu invocation command line would be highly appreciated
> > as well. Thank you!
> >
> Sounds good. However, malta is actually a MIPS platform. I've been using it
> a long time because it makes things particularly easy to switch configuration
> between different word-sizes and endianness.
>
> I had some malta mips images ready to go, but if you need ARM I'll need
> to look into building images for big-endian ARM. Big-endian isn't so common
> in the wild, and I'll need to see if OpenWRT supports these, and how to
> configure with QEMU's 'armvirt' target if possible...
>
> > >
> > > For changes to the Linux build system itself (e.g. pahole endian
> > > options and target endian awareness), you would need to set up a
> >
> > I think that shouldn't be a problem and should be handled
> > transparently, even in a cross-compilation case, but let's see.
> >
> > > standard OpenWRT build environment. I can help with that, or simply
> > > integrate your patches myself for testing. As you say, nothing to be
> > > super pumped about...
> > >
> > > Let me know what's easiest and how best to get images to you.
> >
> > Any way you like and can. Dropbox, Google drive, what have you.
> >
> Meantime, I can package up what I have and send you the details. That
> would include images for mips32/64 big-endian and arm32/64 little-endian,
> plus usage examples. Is that still helpful for you?
>
> Thanks,
> Tony

Re: Problem with endianess of pahole BTF output for vmlinux

[Andrii Nakryiko]

On Mon, Sep 28, 2020 at 11:48 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> On Mon, 28 Sep 2020 at 21:15, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >
> > On Mon, Sep 28, 2020 at 8:41 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > >
> [...]
> > > I can provide 32-bit and 64-bit big-endian system images for running
> > > on QEMU's malta target. These are built using OpenWRT's build system
> > > and include a recent stable bpftool (v5.8.x) and v5.4.x kernel. Is
> > > that sufficient? It would work if manually creating raw or elf-based
> > > BTF files on a build host, then copying into the QEMU target to test
> > > parsing with bpftool (linked with the standard libbpf).
> >
> > That would be great! I intend to run them under qemu-system-arm and
> > supply latest kernel through -kernel option, so kernel itself is not
> > that critical. Same for bpftool, pahole, etc, I'll just supply them
> > from my host environment. So please let me know how I can get ahold of
> > those. Sample qemu invocation command line would be highly appreciated
> > as well. Thank you!
> >
> Sounds good. However, malta is actually a MIPS platform. I've been using it

I don't care about ARM vs MIPS specifically. Needed 32-bit and
big-endian, just to test different variations. MIPS works just fine, I
think.

> a long time because it makes things particularly easy to switch configuration
> between different word-sizes and endianness.
>
> I had some malta mips images ready to go, but if you need ARM I'll need
> to look into building images for big-endian ARM. Big-endian isn't so common
> in the wild, and I'll need to see if OpenWRT supports these, and how to
> configure with QEMU's 'armvirt' target if possible...
>
> > >
> > > For changes to the Linux build system itself (e.g. pahole endian
> > > options and target endian awareness), you would need to set up a
> >
> > I think that shouldn't be a problem and should be handled
> > transparently, even in a cross-compilation case, but let's see.
> >
> > > standard OpenWRT build environment. I can help with that, or simply
> > > integrate your patches myself for testing. As you say, nothing to be
> > > super pumped about...
> > >
> > > Let me know what's easiest and how best to get images to you.
> >
> > Any way you like and can. Dropbox, Google drive, what have you.
> >
> Meantime, I can package up what I have and send you the details. That
> would include images for mips32/64 big-endian and arm32/64 little-endian,
> plus usage examples. Is that still helpful for you?

Yeah, tremendously! After fighting my local qemu setup I managed to
run malta-be image successfully. Thanks a lot, that saves me lots of
gray hair. I'll get to testing this hopefully today-tomorrow, will let
you know if I had any more problems.

>
> Thanks,
> Tony