[Bpf] ISA RFC compliance question

[Bpf] ISA RFC compliance question

[Dave Thaler]

[-- Attachment #1.1: Type: text/plain, Size: 804 bytes --]

Now that we have some new "v4" instructions, it seems a good time to ask about
what it means to support (or comply with) the ISA RFC once published.  Does
it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
non-deprecated instructions in the document?   That is any runtime or tool that
doesn't support the new instructions is considered non-compliant with the BPF ISA?

Or should we create some things that are SHOULDs, or finer grained units of
compliance so as to not declare existing deployments non-compliant?
Previously we only talked about cases where instructions were added in an
extension RFC which would naturally provide a separate RFC to conform to.
But I don't think we discussed things like new instructions in the main spec like
we have now.

Dave

[-- Attachment #1.2: Type: text/html, Size: 2628 bytes --]

[-- Attachment #2: Type: text/plain, Size: 76 bytes --]

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

RE: ISA RFC compliance question

[Dave Thaler]

[fixing weird character issue in email below that caused a bounce]

Now that we have some new "v4" instructions, it seems a good time to ask about
what it means to support (or comply with) the ISA RFC once published.  Does
it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
non-deprecated instructions in the document?   That is any runtime or tool that
doesn't support the new instructions is considered non-compliant with the BPF ISA?

Or should we create some things that are SHOULDs, or finer grained units of
compliance so as to not declare existing deployments non-compliant?
Previously we only talked about cases where instructions were added in an
extension RFC which would naturally provide a separate RFC to conform to.
But I don't think we discussed things like new instructions in the main spec like
we have now.

Dave

Re: [Bpf] ISA RFC compliance question

[Dave Thaler]

[fixing weird character issue in email below that caused a bounce]

Now that we have some new "v4" instructions, it seems a good time to ask about
what it means to support (or comply with) the ISA RFC once published.  Does
it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
non-deprecated instructions in the document?   That is any runtime or tool that
doesn't support the new instructions is considered non-compliant with the BPF ISA?

Or should we create some things that are SHOULDs, or finer grained units of
compliance so as to not declare existing deployments non-compliant?
Previously we only talked about cases where instructions were added in an
extension RFC which would naturally provide a separate RFC to conform to.
But I don't think we discussed things like new instructions in the main spec like
we have now.

Dave

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Alexei Starovoitov]

On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
<dthaler=40microsoft.com@dmarc.ietf.org> wrote:
>
> [fixing weird character issue in email below that caused a bounce]
>
> Now that we have some new "v4" instructions, it seems a good time to ask about
> what it means to support (or comply with) the ISA RFC once published.  Does
> it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
> non-deprecated instructions in the document?   That is any runtime or tool that
> doesn't support the new instructions is considered non-compliant with the BPF ISA?

In the linux kernel not all JITs support all instructions.
That was the case even before v4 additions.
Same goes for various user space tools.

> Or should we create some things that are SHOULDs, or finer grained units of
> compliance so as to not declare existing deployments non-compliant?

I suspect 'non-compliance' label will cause an unnecessary backlash,
so I would go with SHOULD wording.

> Previously we only talked about cases where instructions were added in an
> extension RFC which would naturally provide a separate RFC to conform to.
> But I don't think we discussed things like new instructions in the main spec like
> we have now.
>
> Dave
>
> --
> Bpf mailing list
> Bpf@ietf.org
> https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Alexei Starovoitov]

On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
<dthaler=40microsoft.com@dmarc.ietf.org> wrote:
>
> [fixing weird character issue in email below that caused a bounce]
>
> Now that we have some new "v4" instructions, it seems a good time to ask about
> what it means to support (or comply with) the ISA RFC once published.  Does
> it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
> non-deprecated instructions in the document?   That is any runtime or tool that
> doesn't support the new instructions is considered non-compliant with the BPF ISA?

In the linux kernel not all JITs support all instructions.
That was the case even before v4 additions.
Same goes for various user space tools.

> Or should we create some things that are SHOULDs, or finer grained units of
> compliance so as to not declare existing deployments non-compliant?

I suspect 'non-compliance' label will cause an unnecessary backlash,
so I would go with SHOULD wording.

> Previously we only talked about cases where instructions were added in an
> extension RFC which would naturally provide a separate RFC to conform to.
> But I don't think we discussed things like new instructions in the main spec like
> we have now.
>
> Dave
>
> --
> Bpf mailing list
> Bpf@ietf.org
> https://www.ietf.org/mailman/listinfo/bpf

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

RE: [Bpf] ISA RFC compliance question

[Dave Thaler]

> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: 
> On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
> <dthaler=40microsoft.com@dmarc.ietf.org> wrote:
> > Now that we have some new "v4" instructions, it seems a good time to
> > ask about what it means to support (or comply with) the ISA RFC once
> > published.  Does it mean that a verifier/disassembler/JIT compiler/etc. MUST
> support *all* the
> > non-deprecated instructions in the document?   That is any runtime or tool that
> > doesn't support the new instructions is considered non-compliant with the BPF
> ISA?
[...]
> > Or should we create some things that are SHOULDs, or finer grained
> > units of compliance so as to not declare existing deployments non-compliant?
> 
> I suspect 'non-compliance' label will cause an unnecessary backlash, so I would
> go with SHOULD wording.

Yeah, but if each instruction is a separate SHOULD, then a runtime could (say)
support one atomic instruction and not others.  Having that level of granularity
would really complicate interoperability and cross-platform tooling in my opinion.
So it might be better to list groups of instructions and have the SHOULD be at the
granularity of a group?

Dave

Re: [Bpf] ISA RFC compliance question

[Dave Thaler]

> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: 
> On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
> <dthaler=40microsoft.com@dmarc.ietf.org> wrote:
> > Now that we have some new "v4" instructions, it seems a good time to
> > ask about what it means to support (or comply with) the ISA RFC once
> > published.  Does it mean that a verifier/disassembler/JIT compiler/etc. MUST
> support *all* the
> > non-deprecated instructions in the document?   That is any runtime or tool that
> > doesn't support the new instructions is considered non-compliant with the BPF
> ISA?
[...]
> > Or should we create some things that are SHOULDs, or finer grained
> > units of compliance so as to not declare existing deployments non-compliant?
> 
> I suspect 'non-compliance' label will cause an unnecessary backlash, so I would
> go with SHOULD wording.

Yeah, but if each instruction is a separate SHOULD, then a runtime could (say)
support one atomic instruction and not others.  Having that level of granularity
would really complicate interoperability and cross-platform tooling in my opinion.
So it might be better to list groups of instructions and have the SHOULD be at the
granularity of a group?

Dave
-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Alexei Starovoitov]

On Thu, Oct 5, 2023 at 1:14 PM Dave Thaler <dthaler@microsoft.com> wrote:
>
> > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> > On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
> > <dthaler=40microsoft.com@dmarc.ietf.org> wrote:
> > > Now that we have some new "v4" instructions, it seems a good time to
> > > ask about what it means to support (or comply with) the ISA RFC once
> > > published.  Does it mean that a verifier/disassembler/JIT compiler/etc. MUST
> > support *all* the
> > > non-deprecated instructions in the document?   That is any runtime or tool that
> > > doesn't support the new instructions is considered non-compliant with the BPF
> > ISA?
> [...]
> > > Or should we create some things that are SHOULDs, or finer grained
> > > units of compliance so as to not declare existing deployments non-compliant?
> >
> > I suspect 'non-compliance' label will cause an unnecessary backlash, so I would
> > go with SHOULD wording.
>
> Yeah, but if each instruction is a separate SHOULD, then a runtime could (say)
> support one atomic instruction and not others.  Having that level of granularity
> would really complicate interoperability and cross-platform tooling in my opinion.
> So it might be better to list groups of instructions and have the SHOULD be at the
> granularity of a group?

I guess we can group them based on LLVM evolution of the instruction set:
-mcpu=v1,v2,v3,v4
but it would have mainly historical benefits and not practical.
Grouping atomic vs not is not realistic either, since atomic_xadd
was there since the very beginning.
I suspect any kind of grouping scheme will end up in bike shedding.
My preference would be to agree on either SHOULD or MUST for
all insns currently described in ISA doc.
We can go with MUST to force compliance.
Some archs, OSes, JITs won't be compliant in the short term,
but MUST wording will be a good motivation to do the work now instead of later.

Re: [Bpf] ISA RFC compliance question

[Alexei Starovoitov]

On Thu, Oct 5, 2023 at 1:14 PM Dave Thaler <dthaler@microsoft.com> wrote:
>
> > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> > On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
> > <dthaler=40microsoft.com@dmarc.ietf.org> wrote:
> > > Now that we have some new "v4" instructions, it seems a good time to
> > > ask about what it means to support (or comply with) the ISA RFC once
> > > published.  Does it mean that a verifier/disassembler/JIT compiler/etc. MUST
> > support *all* the
> > > non-deprecated instructions in the document?   That is any runtime or tool that
> > > doesn't support the new instructions is considered non-compliant with the BPF
> > ISA?
> [...]
> > > Or should we create some things that are SHOULDs, or finer grained
> > > units of compliance so as to not declare existing deployments non-compliant?
> >
> > I suspect 'non-compliance' label will cause an unnecessary backlash, so I would
> > go with SHOULD wording.
>
> Yeah, but if each instruction is a separate SHOULD, then a runtime could (say)
> support one atomic instruction and not others.  Having that level of granularity
> would really complicate interoperability and cross-platform tooling in my opinion.
> So it might be better to list groups of instructions and have the SHOULD be at the
> granularity of a group?

I guess we can group them based on LLVM evolution of the instruction set:
-mcpu=v1,v2,v3,v4
but it would have mainly historical benefits and not practical.
Grouping atomic vs not is not realistic either, since atomic_xadd
was there since the very beginning.
I suspect any kind of grouping scheme will end up in bike shedding.
My preference would be to agree on either SHOULD or MUST for
all insns currently described in ISA doc.
We can go with MUST to force compliance.
Some archs, OSes, JITs won't be compliant in the short term,
but MUST wording will be a good motivation to do the work now instead of later.

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Christoph Hellwig]

On Fri, Sep 29, 2023 at 08:14:12PM +0000, Dave Thaler wrote:
> Now that we have some new "v4" instructions, it seems a good time to ask about
> what it means to support (or comply with) the ISA RFC once published.  Does
> it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
> non-deprecated instructions in the document?   That is any runtime or tool that
> doesn't support the new instructions is considered non-compliant with the BPF ISA?

Unless we clearly designate optional extensions that that can clearly
be marked supported or not supported that is the only way to get
interoperability.

Re: [Bpf] ISA RFC compliance question

[Christoph Hellwig]

On Fri, Sep 29, 2023 at 08:14:12PM +0000, Dave Thaler wrote:
> Now that we have some new "v4" instructions, it seems a good time to ask about
> what it means to support (or comply with) the ISA RFC once published.  Does
> it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
> non-deprecated instructions in the document?   That is any runtime or tool that
> doesn't support the new instructions is considered non-compliant with the BPF ISA?

Unless we clearly designate optional extensions that that can clearly
be marked supported or not supported that is the only way to get
interoperability.

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Will Hawkins]

On Thu, Oct 19, 2023 at 2:04 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Fri, Sep 29, 2023 at 08:14:12PM +0000, Dave Thaler wrote:
> > Now that we have some new "v4" instructions, it seems a good time to ask about
> > what it means to support (or comply with) the ISA RFC once published.  Does
> > it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
> > non-deprecated instructions in the document?   That is any runtime or tool that
> > doesn't support the new instructions is considered non-compliant with the BPF ISA?
>
> Unless we clearly designate optional extensions that that can clearly
> be marked supported or not supported that is the only way to get
> interoperability.
>

Can we look to either RISC-V or ARM for prior art in how they worked
different versions and compliance levels? I am happy to amass some
documentation about their processes/procedures if you think that it
would help!

Will


> --
> Bpf mailing list
> Bpf@ietf.org
> https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Will Hawkins]

On Thu, Oct 19, 2023 at 2:04 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Fri, Sep 29, 2023 at 08:14:12PM +0000, Dave Thaler wrote:
> > Now that we have some new "v4" instructions, it seems a good time to ask about
> > what it means to support (or comply with) the ISA RFC once published.  Does
> > it mean that a verifier/disassembler/JIT compiler/etc. MUST support *all* the
> > non-deprecated instructions in the document?   That is any runtime or tool that
> > doesn't support the new instructions is considered non-compliant with the BPF ISA?
>
> Unless we clearly designate optional extensions that that can clearly
> be marked supported or not supported that is the only way to get
> interoperability.
>

Can we look to either RISC-V or ARM for prior art in how they worked
different versions and compliance levels? I am happy to amass some
documentation about their processes/procedures if you think that it
would help!

Will


> --
> Bpf mailing list
> Bpf@ietf.org
> https://www.ietf.org/mailman/listinfo/bpf

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[David Vernet]

On Fri, Oct 06, 2023 at 04:06:53PM -0700, Alexei Starovoitov wrote:
> On Thu, Oct 5, 2023 at 1:14 PM Dave Thaler <dthaler@microsoft.com> wrote:
> >
> > > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> > > On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
> > > <dthaler=40microsoft.com@dmarc.ietf.org> wrote:
> > > > Now that we have some new "v4" instructions, it seems a good time to
> > > > ask about what it means to support (or comply with) the ISA RFC once
> > > > published.  Does it mean that a verifier/disassembler/JIT compiler/etc. MUST
> > > support *all* the
> > > > non-deprecated instructions in the document?   That is any runtime or tool that
> > > > doesn't support the new instructions is considered non-compliant with the BPF
> > > ISA?
> > [...]
> > > > Or should we create some things that are SHOULDs, or finer grained
> > > > units of compliance so as to not declare existing deployments non-compliant?
> > >
> > > I suspect 'non-compliance' label will cause an unnecessary backlash, so I would
> > > go with SHOULD wording.
> >
> > Yeah, but if each instruction is a separate SHOULD, then a runtime could (say)
> > support one atomic instruction and not others.  Having that level of granularity
> > would really complicate interoperability and cross-platform tooling in my opinion.
> > So it might be better to list groups of instructions and have the SHOULD be at the
> > granularity of a group?
> 
> I guess we can group them based on LLVM evolution of the instruction set:
> -mcpu=v1,v2,v3,v4
> but it would have mainly historical benefits and not practical.

We will discuss more at IETF 118, but I agree that grouping based on
LLVM instruction set releases is not a good idea. It's the same
sentiment for why we don't want to standardize the .maps ELF section
just because it's what libbpf expects.

> Grouping atomic vs not is not realistic either, since atomic_xadd
> was there since the very beginning.

This might be a dumb question, but I'm not sure I'm following why it
being introduced since the beginning would preclude it from being
SHOULD?

> I suspect any kind of grouping scheme will end up in bike shedding.
> My preference would be to agree on either SHOULD or MUST for
> all insns currently described in ISA doc.

SHOULD for all instructions seems very risky for compliance. Wouldn't
that functionally make the standard entirely optional?

> We can go with MUST to force compliance.
> Some archs, OSes, JITs won't be compliant in the short term,
> but MUST wording will be a good motivation to do the work now instead of later.

This seems like the overall lowest risk to me, though there are some
nuances we'll have to consider. For example, it would require that all
platforms support BTF in order to be compliant with BPF_CALL by BTF ID.
Realistically that seems desirable. Unless there are groups of
instructions that could be submitted logically as their own separate
extensions, being consistent with MUST seems like the least error prone
approach.

Thanks,
David

Re: [Bpf] ISA RFC compliance question

[David Vernet]

On Fri, Oct 06, 2023 at 04:06:53PM -0700, Alexei Starovoitov wrote:
> On Thu, Oct 5, 2023 at 1:14 PM Dave Thaler <dthaler@microsoft.com> wrote:
> >
> > > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> > > On Fri, Sep 29, 2023 at 1:17 PM Dave Thaler
> > > <dthaler=40microsoft.com@dmarc.ietf.org> wrote:
> > > > Now that we have some new "v4" instructions, it seems a good time to
> > > > ask about what it means to support (or comply with) the ISA RFC once
> > > > published.  Does it mean that a verifier/disassembler/JIT compiler/etc. MUST
> > > support *all* the
> > > > non-deprecated instructions in the document?   That is any runtime or tool that
> > > > doesn't support the new instructions is considered non-compliant with the BPF
> > > ISA?
> > [...]
> > > > Or should we create some things that are SHOULDs, or finer grained
> > > > units of compliance so as to not declare existing deployments non-compliant?
> > >
> > > I suspect 'non-compliance' label will cause an unnecessary backlash, so I would
> > > go with SHOULD wording.
> >
> > Yeah, but if each instruction is a separate SHOULD, then a runtime could (say)
> > support one atomic instruction and not others.  Having that level of granularity
> > would really complicate interoperability and cross-platform tooling in my opinion.
> > So it might be better to list groups of instructions and have the SHOULD be at the
> > granularity of a group?
> 
> I guess we can group them based on LLVM evolution of the instruction set:
> -mcpu=v1,v2,v3,v4
> but it would have mainly historical benefits and not practical.

We will discuss more at IETF 118, but I agree that grouping based on
LLVM instruction set releases is not a good idea. It's the same
sentiment for why we don't want to standardize the .maps ELF section
just because it's what libbpf expects.

> Grouping atomic vs not is not realistic either, since atomic_xadd
> was there since the very beginning.

This might be a dumb question, but I'm not sure I'm following why it
being introduced since the beginning would preclude it from being
SHOULD?

> I suspect any kind of grouping scheme will end up in bike shedding.
> My preference would be to agree on either SHOULD or MUST for
> all insns currently described in ISA doc.

SHOULD for all instructions seems very risky for compliance. Wouldn't
that functionally make the standard entirely optional?

> We can go with MUST to force compliance.
> Some archs, OSes, JITs won't be compliant in the short term,
> but MUST wording will be a good motivation to do the work now instead of later.

This seems like the overall lowest risk to me, though there are some
nuances we'll have to consider. For example, it would require that all
platforms support BTF in order to be compliant with BPF_CALL by BTF ID.
Realistically that seems desirable. Unless there are groups of
instructions that could be submitted logically as their own separate
extensions, being consistent with MUST seems like the least error prone
approach.

Thanks,
David

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf

Re: [Bpf] ISA RFC compliance question

[Christoph Hellwig]

On Mon, Oct 23, 2023 at 11:06:32AM -0400, Will Hawkins wrote:
> Can we look to either RISC-V or ARM for prior art in how they worked
> different versions and compliance levels? I am happy to amass some
> documentation about their processes/procedures if you think that it
> would help!

I don't know arm to well.  x86 is mostly discoverable by bits set in the
cpuid leaves, and RISC-V has formal extensions, initially letter and
after they noticed that isn't fine grained enough a lot of Z-prefixed
ones.  I think RISC-V is a good starting point:

  https://en.wikichip.org/wiki/risc-v/standard_extensions

we need to be very careful on what are good scopes for extensions so
they are on the one hand useful, but on the other hand not too many
of them so that we stop being easily interoperable.

As seen in the other branch of this discussion we should also thing
a bit of what is supported by existing implementations.  I don't think
it is a non-startert if they become non-compliant, but we need to weight
this very carefully.

Re: [Bpf] ISA RFC compliance question

[Christoph Hellwig]

On Mon, Oct 23, 2023 at 11:06:32AM -0400, Will Hawkins wrote:
> Can we look to either RISC-V or ARM for prior art in how they worked
> different versions and compliance levels? I am happy to amass some
> documentation about their processes/procedures if you think that it
> would help!

I don't know arm to well.  x86 is mostly discoverable by bits set in the
cpuid leaves, and RISC-V has formal extensions, initially letter and
after they noticed that isn't fine grained enough a lot of Z-prefixed
ones.  I think RISC-V is a good starting point:

  https://en.wikichip.org/wiki/risc-v/standard_extensions

we need to be very careful on what are good scopes for extensions so
they are on the one hand useful, but on the other hand not too many
of them so that we stop being easily interoperable.

As seen in the other branch of this discussion we should also thing
a bit of what is supported by existing implementations.  I don't think
it is a non-startert if they become non-compliant, but we need to weight
this very carefully.

-- 
Bpf mailing list
Bpf@ietf.org
https://www.ietf.org/mailman/listinfo/bpf