Re: [PATCH 0/9] eBPF support for GNU binutils

Re: [PATCH 0/9] eBPF support for GNU binutils

[Jiong Wang]

CCing BPF kernel community who is defining the ISA and various runtime stuff.

Also two inline comments below about the assembler

> On 20 May 2019, at 17:45, Jose E. Marchesi <jose.marchesi oracle ! com> wrote:
> 
> Hi people!
> 
> This patch series introduces support for eBPF, which is a virtual
> machine that resides in the Linux kernel.  Initially intended for
> user-level packet capture and filtering, eBPF is nowadays generalized
> to serve as a general-purpose infrastructure also for non-networking
> purposes.
> 
> The first patch is preparatory, and adds support to config.guess to
> recognize bpf-*-* triplets.  This will be submitted as a patch to the
> `config' project as soon as this series gets upstreamed.
> 
> The second and third patches add support for an ELF64 based eBPF
> target to BFD, in both little-endian and big-endian vectors.
> 
> The fourth patch adds a CGEN cpu description for eBPF, plus support
> code.  This description covers the full eBPF ISA.  Due to the 64-bit
> instruction fields used in some instructions, we needed to fix a
> bug/limitation in CGEN impacting 32-bit hosts.  The fix is in a patch
> submitted to CGEN last week, that is still waiting for review:
> http://www.sourceware.org/ml/cgen/2019-q2/msg00008.html None of the
> existing CGEN ports in binutils are impacted by that patch: the code
> generated for these remains exactly the same.
> 
> The fifth patch adds opcodes and disassembler support for eBPF, based
> on the CGEN description.
> 
> The sixth patch adds a GAS port, including a testsuite and manual
> updates.  By default the assembler generates objects using the same
> endianness than the host.  This can be overrided by the usual -EB and
> -EB command-line options.
> 
> Support for linking eBPF ELF files with ld/bfd is provided in the
> seventh patch.  A couple of simple tests are included.
> 
> The eighth patch adds support for eBPF to readelf, and makes a little
> adjustment in the `nm' testsuite to not fail in bpf-*-* targets.
> 
> Finally, the last patch adds myself as the maintainer of the BPF
> target.  We are committing to maintain this port.
> 
> Future work on the binutils port:
> * Support for semantic actions in bpf.cpu, and support code for a
>  simulator in sim/.
> * Support for ld.gold.
> 
> Next stop is GCC.  An eBPF backend is on the works.  We plan to
> upstream it before September.
> 
> Regressions tested in all targets.
> Regressions tested with --enable-targets=all
> Tested in 64-bit x86_64 host.
> Tested in 32-bit x86 host.
> 
> Oh, a little note regarding interoperability:
> 
> There is a clang/llvm based toolchain for eBPF.  However, at this
> moment compiled eBPF doesn't have established conventions.  The
> details on what is expected to be in an ELF file containing eBPF is
> determined, in practice, by what the llvm BPF backend supports and
> what the sample bpf_load.c in the Linux kernel source tree expects
> [1].
> 
> Despite using a different syntax for the assembler (the llvm assembler
> uses a C-ish expression-based syntax while the GNU assembler opts for
> a more classic assembly-language syntax) this implementation tries to
> provide inter-operability with clang/llvm generated objects.

I also noticed your implementation doesn’t seem to use the same sub-register
syntax as what LLVM assembler is doing.

  x register for 64-bit, and w register for 32-bit sub-register.

So:
  add r0, r1, r2 means BPF_ALU64 | BPF_ADD | BFF_X
  add w0, w1, w1 means BPF_ALU | BPF_ADD | BPF_X

ASAICT, different register prefix for different register width is also adopted
by quite a few other GNU assembler targets like AArch64, X86_64.

> 
> In particular, the numbers of the relocations used for instruction
> fields are the same.  These are R_BPF_INSN_64 and R_BPF_INSN_DISP32.
> The later is resolved at load-time by bpf_load.c.

I think you missed the latest JMP32 instructions.

  https://github.com/torvalds/linux/blob/master/Documentation/networking/filter.txt#L870


> 
> [1] We expect/hope that the addition of eBPF support to the GNU
>    toolchain will help to mature the domain of compiled eBPF.  We
>    will certainly be working with the kernel people to that effect.
> 
> Salud!
> 
> Jose E. Marchesi (9):
>  config: recognize eBPF triplets
>  include: add elf/bpf.h
>  bfd: add support for eBPF
>  cpu: add eBPF cpu description
>  opcodes: add support for eBPF
>  gas: add support for eBPF
>  ld: add support for eBPF
>  binutils: add support for eBPF
>  binutils: add myself as the maintainer for BPF
> 
> ChangeLog                              |    4 +
> bfd/ChangeLog                          |   20 +
> bfd/Makefile.am                        |    4 +
> bfd/Makefile.in                        |    7 +
> bfd/archures.c                         |    4 +
> bfd/bfd-in2.h                          |    9 +
> bfd/config.bfd                         |    6 +
> bfd/configure                          |   30 +-
> bfd/configure.ac                       |    2 +
> bfd/cpu-bpf.c                          |   41 +
> bfd/elf64-bpf.c                        |  463 +++++++++
> bfd/libbfd.h                           |    5 +
> bfd/reloc.c                            |   13 +
> bfd/targets.c                          |    7 +
> binutils/ChangeLog                     |   13 +
> binutils/MAINTAINERS                   |    1 +
> binutils/readelf.c                     |    8 +
> binutils/testsuite/binutils-all/nm.exp |    3 +-
> config.sub                             |    4 +-
> cpu/ChangeLog                          |    5 +
> cpu/bpf.cpu                            |  647 +++++++++++++
> cpu/bpf.opc                            |  191 ++++
> gas/ChangeLog                          |   45 +
> gas/Makefile.am                        |    2 +
> gas/Makefile.in                        |    6 +
> gas/config/tc-bpf.c                    |  357 +++++++
> gas/config/tc-bpf.h                    |   51 +
> gas/configure                          |   38 +-
> gas/configure.ac                       |    6 +
> gas/configure.tgt                      |    1 +
> gas/doc/Makefile.am                    |    1 +
> gas/doc/Makefile.in                    |    6 +-
> gas/doc/all.texi                       |    1 +
> gas/doc/as.texi                        |   34 +
> gas/doc/c-bpf.texi                     |  364 +++++++
> gas/testsuite/gas/all/gas.exp          |    3 +
> gas/testsuite/gas/all/org-1.l          |    2 +-
> gas/testsuite/gas/all/org-1.s          |    2 +
> gas/testsuite/gas/bpf/alu-be.d         |   59 ++
> gas/testsuite/gas/bpf/alu.d            |   58 ++
> gas/testsuite/gas/bpf/alu.s            |   51 +
> gas/testsuite/gas/bpf/alu32-be.d       |   65 ++
> gas/testsuite/gas/bpf/alu32.d          |   64 ++
> gas/testsuite/gas/bpf/alu32.s          |   57 ++
> gas/testsuite/gas/bpf/atomic-be.d      |   12 +
> gas/testsuite/gas/bpf/atomic.d         |   11 +
> gas/testsuite/gas/bpf/atomic.s         |    5 +
> gas/testsuite/gas/bpf/bpf.exp          |   38 +
> gas/testsuite/gas/bpf/call-be.d        |   19 +
> gas/testsuite/gas/bpf/call.d           |   18 +
> gas/testsuite/gas/bpf/call.s           |   11 +
> gas/testsuite/gas/bpf/exit-be.d        |   11 +
> gas/testsuite/gas/bpf/exit.d           |   10 +
> gas/testsuite/gas/bpf/exit.s           |    2 +
> gas/testsuite/gas/bpf/jump-be.d        |   32 +
> gas/testsuite/gas/bpf/jump.d           |   31 +
> gas/testsuite/gas/bpf/jump.s           |   25 +
> gas/testsuite/gas/bpf/lddw-be.d        |   18 +
> gas/testsuite/gas/bpf/lddw.d           |   17 +
> gas/testsuite/gas/bpf/lddw.s           |    6 +
> gas/testsuite/gas/bpf/mem-be.d         |   30 +
> gas/testsuite/gas/bpf/mem.d            |   29 +
> gas/testsuite/gas/bpf/mem.s            |   24 +
> include/ChangeLog                      |    4 +
> include/elf/bpf.h                      |   45 +
> ld/ChangeLog                           |   15 +
> ld/Makefile.am                         |    2 +
> ld/Makefile.in                         |    4 +
> ld/configure                           |   28 +-
> ld/configure.tgt                       |    1 +
> ld/emulparams/elf64bpf.sh              |   10 +
> ld/testsuite/ld-bpf/bar.s              |    5 +
> ld/testsuite/ld-bpf/baz.s              |    5 +
> ld/testsuite/ld-bpf/bpf.exp            |   29 +
> ld/testsuite/ld-bpf/call-1.d           |   23 +
> ld/testsuite/ld-bpf/foo.s              |    5 +
> ld/testsuite/ld-bpf/jump-1.d           |   23 +
> ld/testsuite/lib/ld-lib.exp            |    1 +
> opcodes/ChangeLog                      |   24 +
> opcodes/Makefile.am                    |   17 +
> opcodes/Makefile.in                    |   23 +
> opcodes/bpf-asm.c                      |  590 ++++++++++++
> opcodes/bpf-desc.c                     | 1638 ++++++++++++++++++++++++++++++++
> opcodes/bpf-desc.h                     |  266 ++++++
> opcodes/bpf-dis.c                      |  624 ++++++++++++
> opcodes/bpf-ibld.c                     |  956 +++++++++++++++++++
> opcodes/bpf-opc.c                      | 1495 +++++++++++++++++++++++++++++
> opcodes/bpf-opc.h                      |  151 +++
> opcodes/configure                      |   19 +-
> opcodes/configure.ac                   |    1 +
> opcodes/disassemble.c                  |   35 +
> opcodes/disassemble.h                  |    1 +
> 92 files changed, 9116 insertions(+), 33 deletions(-)
> create mode 100644 bfd/cpu-bpf.c
> create mode 100644 bfd/elf64-bpf.c
> create mode 100644 cpu/bpf.cpu
> create mode 100644 cpu/bpf.opc
> create mode 100644 gas/config/tc-bpf.c
> create mode 100644 gas/config/tc-bpf.h
> create mode 100644 gas/doc/c-bpf.texi
> create mode 100644 gas/testsuite/gas/bpf/alu-be.d
> create mode 100644 gas/testsuite/gas/bpf/alu.d
> create mode 100644 gas/testsuite/gas/bpf/alu.s
> create mode 100644 gas/testsuite/gas/bpf/alu32-be.d
> create mode 100644 gas/testsuite/gas/bpf/alu32.d
> create mode 100644 gas/testsuite/gas/bpf/alu32.s
> create mode 100644 gas/testsuite/gas/bpf/atomic-be.d
> create mode 100644 gas/testsuite/gas/bpf/atomic.d
> create mode 100644 gas/testsuite/gas/bpf/atomic.s
> create mode 100644 gas/testsuite/gas/bpf/bpf.exp
> create mode 100644 gas/testsuite/gas/bpf/call-be.d
> create mode 100644 gas/testsuite/gas/bpf/call.d
> create mode 100644 gas/testsuite/gas/bpf/call.s
> create mode 100644 gas/testsuite/gas/bpf/exit-be.d
> create mode 100644 gas/testsuite/gas/bpf/exit.d
> create mode 100644 gas/testsuite/gas/bpf/exit.s
> create mode 100644 gas/testsuite/gas/bpf/jump-be.d
> create mode 100644 gas/testsuite/gas/bpf/jump.d
> create mode 100644 gas/testsuite/gas/bpf/jump.s
> create mode 100644 gas/testsuite/gas/bpf/lddw-be.d
> create mode 100644 gas/testsuite/gas/bpf/lddw.d
> create mode 100644 gas/testsuite/gas/bpf/lddw.s
> create mode 100644 gas/testsuite/gas/bpf/mem-be.d
> create mode 100644 gas/testsuite/gas/bpf/mem.d
> create mode 100644 gas/testsuite/gas/bpf/mem.s
> create mode 100644 include/elf/bpf.h
> create mode 100644 ld/emulparams/elf64bpf.sh
> create mode 100644 ld/testsuite/ld-bpf/bar.s
> create mode 100644 ld/testsuite/ld-bpf/baz.s
> create mode 100644 ld/testsuite/ld-bpf/bpf.exp
> create mode 100644 ld/testsuite/ld-bpf/call-1.d
> create mode 100644 ld/testsuite/ld-bpf/foo.s
> create mode 100644 ld/testsuite/ld-bpf/jump-1.d
> create mode 100644 opcodes/bpf-asm.c
> create mode 100644 opcodes/bpf-desc.c
> create mode 100644 opcodes/bpf-desc.h
> create mode 100644 opcodes/bpf-dis.c
> create mode 100644 opcodes/bpf-ibld.c
> create mode 100644 opcodes/bpf-opc.c
> create mode 100644 opcodes/bpf-opc.h
> 
> -- 
> 2.11.0

Re: [PATCH 0/9] eBPF support for GNU binutils

[Jose E. Marchesi]

Hi Jiong.

    > Despite using a different syntax for the assembler (the llvm assembler
    > uses a C-ish expression-based syntax while the GNU assembler opts for
    > a more classic assembly-language syntax) this implementation tries to
    > provide inter-operability with clang/llvm generated objects.
    
    I also noticed your implementation doesn’t seem to use the same sub-register
    syntax as what LLVM assembler is doing.
    
      x register for 64-bit, and w register for 32-bit sub-register.
    
    So:
      add r0, r1, r2 means BPF_ALU64 | BPF_ADD | BFF_X
      add w0, w1, w1 means BPF_ALU | BPF_ADD | BPF_X
    
    ASAICT, different register prefix for different register width is also adopted
    by quite a few other GNU assembler targets like AArch64, X86_64.

Right.  I opted for using different mnemonics for alu and alu64
instructions, as it seemed to be simpler.

What was your rationale for using sub-register notation?  Are you
planning to support instructions (or pseudo-instructions) mixing w and x
registers in the future?

    > In particular, the numbers of the relocations used for instruction
    > fields are the same.  These are R_BPF_INSN_64 and R_BPF_INSN_DISP32.
    > The later is resolved at load-time by bpf_load.c.
    
    I think you missed the latest JMP32 instructions.
    
      https://github.com/torvalds/linux/blob/master/Documentation/networking/filter.txt#L870

Oh thanks for spotting that.
Adding support for it :)

Re: [PATCH 0/9] eBPF support for GNU binutils

[David Miller]

From: Jiong Wang <jiong.wang@netronome.com>
Date: Tue, 21 May 2019 16:41:56 +0100

> CCing BPF kernel community who is defining the ISA and various runtime stuff.
> 
> Also two inline comments below about the assembler

Also, I already wrote such a binutils patch a year or two ago:

https://lwn.net/Articles/721473/

Re: [PATCH 0/9] eBPF support for GNU binutils

[Jiong Wang]

> On 21 May 2019, at 18:06, Jose E. Marchesi <jose.marchesi@oracle.com> wrote:
> 
> 
> Hi Jiong.
> 
>> Despite using a different syntax for the assembler (the llvm assembler
>> uses a C-ish expression-based syntax while the GNU assembler opts for
>> a more classic assembly-language syntax) this implementation tries to
>> provide inter-operability with clang/llvm generated objects.
> 
>    I also noticed your implementation doesn’t seem to use the same sub-register
>    syntax as what LLVM assembler is doing.
> 
>      x register for 64-bit, and w register for 32-bit sub-register.
> 
>    So:
>      add r0, r1, r2 means BPF_ALU64 | BPF_ADD | BFF_X
>      add w0, w1, w1 means BPF_ALU | BPF_ADD | BPF_X
> 
>    ASAICT, different register prefix for different register width is also adopted
>    by quite a few other GNU assembler targets like AArch64, X86_64.
> 
> Right.  I opted for using different mnemonics for alu and alu64
> instructions, as it seemed to be simpler.
> 
> What was your rationale for using sub-register notation?  

It is the same instruction operating on different register classes, sub-register
is a new register class, so define separate notation for them. This also
simplifies compiler back-end when generating sub-register instructions, at
least for LLVM, and is likely for GCC as well. 

LLVM eBPF backend has full support for generating sub-register ISA,


> Are you
> planning to support instructions (or pseudo-instructions) mixing w and x
> registers in the future?
> 
>> In particular, the numbers of the relocations used for instruction
>> fields are the same.  These are R_BPF_INSN_64 and R_BPF_INSN_DISP32.
>> The later is resolved at load-time by bpf_load.c.
> 
>    I think you missed the latest JMP32 instructions.
> 
>      https://github.com/torvalds/linux/blob/master/Documentation/networking/filter.txt#L870
> 
> Oh thanks for spotting that.
> Adding support for it :)

Re: [PATCH 0/9] eBPF support for GNU binutils

[Alexei Starovoitov]

On Tue, May 21, 2019 at 8:41 AM Jiong Wang <jiong.wang@netronome.com> wrote:
> >
> > Despite using a different syntax for the assembler (the llvm assembler
> > uses a C-ish expression-based syntax while the GNU assembler opts for
> > a more classic assembly-language syntax) this implementation tries to
> > provide inter-operability with clang/llvm generated objects.
>
> I also noticed your implementation doesn’t seem to use the same sub-register
> syntax as what LLVM assembler is doing.
>
>   x register for 64-bit, and w register for 32-bit sub-register.
>
> So:
>   add r0, r1, r2 means BPF_ALU64 | BPF_ADD | BFF_X
>   add w0, w1, w1 means BPF_ALU | BPF_ADD | BPF_X
>
> ASAICT, different register prefix for different register width is also adopted
> by quite a few other GNU assembler targets like AArch64, X86_64.

there is also Ed's assembler:
https://github.com/solarflarecom/ebpf_asm
It uses 2 ops style.
I think 3 ops style "add r0,r1,r2" is not a good fit for bpf isa.

I think we need to converge on one asm syntax for gas/bfd.
At this point we cannot change llvm's asm output,
so my preference would be to make gas accept it.
But I understand the implementation difficulties to fit it into bfd infra.
So I'm ok with more traditional asm the way Dave implemented it few
years back.
One asm syntax for gas and another asm syntax for clang is, imo, acceptable.

Jose, can you combine Dave's patches with yours?

I think Ed had an idea on how to specify BTF in asm syntax.
BTF has to be supported by the assembler as well
along with .btf.ext, lineinfo, etc
Currently llvm emits btf as '.byte 0x...', but that's far from ideal.

Re: [PATCH 0/9] eBPF support for GNU binutils

[Jose E. Marchesi]

Hi Alexey.

    > > Despite using a different syntax for the assembler (the llvm assembler
    > > uses a C-ish expression-based syntax while the GNU assembler opts for
    > > a more classic assembly-language syntax) this implementation tries to
    > > provide inter-operability with clang/llvm generated objects.
    >
    > I also noticed your implementation doesn’t seem to use the same sub-register
    > syntax as what LLVM assembler is doing.
    >
    >   x register for 64-bit, and w register for 32-bit sub-register.
    >
    > So:
    >   add r0, r1, r2 means BPF_ALU64 | BPF_ADD | BFF_X
    >   add w0, w1, w1 means BPF_ALU | BPF_ADD | BPF_X
    >
    > ASAICT, different register prefix for different register width is also adopted
    > by quite a few other GNU assembler targets like AArch64, X86_64.
    
    there is also Ed's assembler:
    https://github.com/solarflarecom/ebpf_asm

Oh interesting, didn't know about it.

    It uses 2 ops style.
    I think 3 ops style "add r0,r1,r2" is not a good fit for bpf isa.

I agree.  My port uses two-operand formats for most instructions,
including arithmetic instructions.  Examples:

sub %r2, 666
sub %r3, -666
sub %r4, 0x7eadbeef
sub %r5, %r6
    
    I think we need to converge on one asm syntax for gas/bfd.
    At this point we cannot change llvm's asm output,
    so my preference would be to make gas accept it.
    But I understand the implementation difficulties to fit it into bfd infra.
    So I'm ok with more traditional asm the way Dave implemented it few
    years back.
    One asm syntax for gas and another asm syntax for clang is, imo, acceptable.
    
    Jose, can you combine Dave's patches with yours?

As far as I can see my port uses basically the same syntax than Dave's.

The only significant difference is that mine uses a % prefix for
register names.  I prefer it this way in order to avoid collisions with
symbols.
    
    I think Ed had an idea on how to specify BTF in asm syntax.
    BTF has to be supported by the assembler as well
    along with .btf.ext, lineinfo, etc
    Currently llvm emits btf as '.byte 0x...', but that's far from ideal.

I haven't looked into BTF yet, but it would be certainly good, for
everyone to use the same directives to support it :)

Re: [PATCH 0/9] eBPF support for GNU binutils

[Edward Cree]

On 21/05/2019 19:18, Alexei Starovoitov wrote:
> I think Ed had an idea on how to specify BTF in asm syntax.
Specifically, see [1] for BTF as implemented in ebpf_asm, though note
 that it doesn't (yet) cover .btf.ext or lineinfo.

-Ed

[1]: https://github.com/solarflarecom/ebpf_asm/tree/btfdoc#type-definitions

Re: [PATCH 0/9] eBPF support for GNU binutils

[Jose E. Marchesi]

    >> Despite using a different syntax for the assembler (the llvm assembler
    >> uses a C-ish expression-based syntax while the GNU assembler opts for
    >> a more classic assembly-language syntax) this implementation tries to
    >> provide inter-operability with clang/llvm generated objects.
    > 
    >    I also noticed your implementation doesn’t seem to use the same sub-register
    >    syntax as what LLVM assembler is doing.
    > 
    >      x register for 64-bit, and w register for 32-bit sub-register.
    > 
    >    So:
    >      add r0, r1, r2 means BPF_ALU64 | BPF_ADD | BFF_X
    >      add w0, w1, w1 means BPF_ALU | BPF_ADD | BPF_X
    > 
    >    ASAICT, different register prefix for different register width is also adopted
    >    by quite a few other GNU assembler targets like AArch64, X86_64.
    > 
    > Right.  I opted for using different mnemonics for alu and alu64
    > instructions, as it seemed to be simpler.
    > 
    > What was your rationale for using sub-register notation?  
    
    It is the same instruction operating on different register classes,
    sub-register is a new register class, so define separate notation
    for them. This also simplifies compiler back-end when generating
    sub-register instructions, at least for LLVM, and is likely for GCC
    as well.

    LLVM eBPF backend has full support for generating sub-register ISA,

Well, the way I read the spec, these look like different instructions
operating on the same registers, only with different semantics :)

But yeah, it is basically two different ways to look at the same thing,
at the ISA level.

Given that both llvm and ebpf_asm use some kind of sub-registers (using
different register names, or suffixes) I guess I could do the same... In
principle I don't have a strong preference, but I have to think about
it, and determine what would be the impact in my on-going GCC backend.

Thanks for the info.  Much appreciated.

Re: [PATCH 0/9] eBPF support for GNU binutils

[Jose E. Marchesi]

    On 21/05/2019 19:18, Alexei Starovoitov wrote:
    > I think Ed had an idea on how to specify BTF in asm syntax.
    Specifically, see [1] for BTF as implemented in ebpf_asm, though note
    that it doesn't (yet) cover .btf.ext or lineinfo.
    
    [1]: https://github.com/solarflarecom/ebpf_asm/tree/btfdoc#type-definitions

Thanks for the reference.  I just checked out your `btfdoc' branch.  I
will take a look.

Where would you like to get feedback/suggestions/questions btw?
bpf@vger.kernel.org?

Re: [PATCH 0/9] eBPF support for GNU binutils

[Edward Cree]

On 21/05/2019 20:34, Jose E. Marchesi wrote:
>     On 21/05/2019 19:18, Alexei Starovoitov wrote:
>     > I think Ed had an idea on how to specify BTF in asm syntax.
>     Specifically, see [1] for BTF as implemented in ebpf_asm, though note
>     that it doesn't (yet) cover .btf.ext or lineinfo.
>     
>     [1]: https://github.com/solarflarecom/ebpf_asm/tree/btfdoc#type-definitions
>
> Thanks for the reference.  I just checked out your `btfdoc' branch.  I
> will take a look.
>
> Where would you like to get feedback/suggestions/questions btw?
For ebpf_asm?  Either through GitHub Issues, or this address (ecree@solarflare.com).

I'm not actually on bpf@vger (though maybe I ought to subscribe...)

BTW I'm really happy to see someone working on eBPF support in the GNU
 toolchain; being tied to an LLVM monoculture has been bad for eBPF imho.

-Ed