Query on new ethtool RSS hashing options

Query on new ethtool RSS hashing options

[Sunil Kovvuri]

Hi,

We have a requirement where in we want RSS hashing to be done on packet fields
which are not currently supported by the ethtool.

Current options:
ehtool -n <dev> rx-flow-hash
tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r

Specifically our requirement is to calculate hash with DSA tag (which
is inserted by switch) plus the TCP/UDP 4-tuple as input.

Is it okay to add such options to the ethtool ?
or will it be better to add a generic option to take pkt data offset
and number of bytes ?

Something like
ethtool -n <dev> rx-flow-hash tcp4 sdfn off <offset in the pkt> num
<number of bytes/bits>

Any comments, please.

Thanks,
Sunil.

Re: Query on new ethtool RSS hashing options

[Jakub Kicinski]

On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> Hi,
> 
> We have a requirement where in we want RSS hashing to be done on packet fields
> which are not currently supported by the ethtool.
> 
> Current options:
> ehtool -n <dev> rx-flow-hash
> tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> 
> Specifically our requirement is to calculate hash with DSA tag (which
> is inserted by switch) plus the TCP/UDP 4-tuple as input.

Can you share the format of the DSA tag? Is there a driver for it
upstream? Do we need to represent it in union ethtool_flow_union?

> Is it okay to add such options to the ethtool ?
> or will it be better to add a generic option to take pkt data offset
> and number of bytes ?
> 
> Something like
> ethtool -n <dev> rx-flow-hash tcp4 sdfn off <offset in the pkt> num
> <number of bytes/bits>
> 
> Any comments, please.

Re: Query on new ethtool RSS hashing options

[Andrew Lunn]

On Fri, Mar 05, 2021 at 03:07:02PM -0800, Jakub Kicinski wrote:
> On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > Hi,
> > 
> > We have a requirement where in we want RSS hashing to be done on packet fields
> > which are not currently supported by the ethtool.
> > 
> > Current options:
> > ehtool -n <dev> rx-flow-hash
> > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> > 
> > Specifically our requirement is to calculate hash with DSA tag (which
> > is inserted by switch) plus the TCP/UDP 4-tuple as input.
> 
> Can you share the format of the DSA tag? Is there a driver for it
> upstream? Do we need to represent it in union ethtool_flow_union?

Sorry, i missed the original question, there was no hint in the
subject line that DSA was involved.

Why do you want to include DSA tag in the hash? What normally happens
with DSA tag drivers is we detect the frame has been received from a
switch, and modify where the core flow dissect code looks in the frame
to skip over the DSA header and parse the IP header etc as normal.

Take a look at net/core/flow_dissect.c:__skb_flow_dissect()

This fits with the core ideas of the network stack and offloads. Hide
the fact an offload is being used, it should just look like a normal
interface.

	Andrew

Re: Query on new ethtool RSS hashing options

[Sunil Kovvuri]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

On Sat, Mar 6, 2021 at 4:37 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > Hi,
> >
> > We have a requirement where in we want RSS hashing to be done on packet fields
> > which are not currently supported by the ethtool.
> >
> > Current options:
> > ehtool -n <dev> rx-flow-hash
> > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> >
> > Specifically our requirement is to calculate hash with DSA tag (which
> > is inserted by switch) plus the TCP/UDP 4-tuple as input.
>
> Can you share the format of the DSA tag? Is there a driver for it
> upstream? Do we need to represent it in union ethtool_flow_union?
>

No, there is no driver for this tag in the kernel.
I have attached the tag format.
There are multiple DSA tag formats and representing them ethtool_flow
union would be difficult.
Hence wondering if it would be okay to add a more flexible way ie
offset and num_bytes from the start of packet.

Thanks,
Sunil.

[-- Attachment #2: FDSA_tag_format.jpg --]
[-- Type: image/jpeg, Size: 151135 bytes --]

Re: Query on new ethtool RSS hashing options

[Sunil Kovvuri]

On Sat, Mar 6, 2021 at 5:47 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Fri, Mar 05, 2021 at 03:07:02PM -0800, Jakub Kicinski wrote:
> > On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > > Hi,
> > >
> > > We have a requirement where in we want RSS hashing to be done on packet fields
> > > which are not currently supported by the ethtool.
> > >
> > > Current options:
> > > ehtool -n <dev> rx-flow-hash
> > > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> > >
> > > Specifically our requirement is to calculate hash with DSA tag (which
> > > is inserted by switch) plus the TCP/UDP 4-tuple as input.
> >
> > Can you share the format of the DSA tag? Is there a driver for it
> > upstream? Do we need to represent it in union ethtool_flow_union?
>
> Sorry, i missed the original question, there was no hint in the
> subject line that DSA was involved.
>
> Why do you want to include DSA tag in the hash? What normally happens
> with DSA tag drivers is we detect the frame has been received from a
> switch, and modify where the core flow dissect code looks in the frame
> to skip over the DSA header and parse the IP header etc as normal.

I understand your point.
The requirement to add DSA tag into RSS hashing is coming from one of
our customer.

>
> Take a look at net/core/flow_dissect.c:__skb_flow_dissect()
>
> This fits with the core ideas of the network stack and offloads. Hide
> the fact an offload is being used, it should just look like a normal
> interface.
>
>         Andrew

Yes, the pkt will look like a normal packet itself.
In our case HW strips the DSA tag from the packet and forwards it to a VF.

Thanks,
Sunil.

Re: Query on new ethtool RSS hashing options

[Vladimir Oltean]

On Sat, Mar 06, 2021 at 05:38:14PM +0530, Sunil Kovvuri wrote:
> > Can you share the format of the DSA tag? Is there a driver for it
> > upstream? Do we need to represent it in union ethtool_flow_union?
> >
> 
> No, there is no driver for this tag in the kernel.
> I have attached the tag format.
> There are multiple DSA tag formats and representing them ethtool_flow
> union would be difficult.
> Hence wondering if it would be okay to add a more flexible way ie
> offset and num_bytes from the start of packet.

How sure are you that the tag format you've shared is not identical to
the one parsed by net/dsa/tag_dsa.c?

Re: Query on new ethtool RSS hashing options

[Vladimir Oltean]

On Sat, Mar 06, 2021 at 06:04:14PM +0530, Sunil Kovvuri wrote:
> On Sat, Mar 6, 2021 at 5:47 AM Andrew Lunn <andrew@lunn.ch> wrote:
> >
> > On Fri, Mar 05, 2021 at 03:07:02PM -0800, Jakub Kicinski wrote:
> > > On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > > > Hi,
> > > >
> > > > We have a requirement where in we want RSS hashing to be done on packet fields
> > > > which are not currently supported by the ethtool.
> > > >
> > > > Current options:
> > > > ehtool -n <dev> rx-flow-hash
> > > > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> > > >
> > > > Specifically our requirement is to calculate hash with DSA tag (which
> > > > is inserted by switch) plus the TCP/UDP 4-tuple as input.
> > >
> > > Can you share the format of the DSA tag? Is there a driver for it
> > > upstream? Do we need to represent it in union ethtool_flow_union?
> >
> > Sorry, i missed the original question, there was no hint in the
> > subject line that DSA was involved.
> >
> > Why do you want to include DSA tag in the hash? What normally happens
> > with DSA tag drivers is we detect the frame has been received from a
> > switch, and modify where the core flow dissect code looks in the frame
> > to skip over the DSA header and parse the IP header etc as normal.
>
> I understand your point.
> The requirement to add DSA tag into RSS hashing is coming from one of
> our customer.
>
> >
> > Take a look at net/core/flow_dissect.c:__skb_flow_dissect()
> >
> > This fits with the core ideas of the network stack and offloads. Hide
> > the fact an offload is being used, it should just look like a normal
> > interface.
> >
> >         Andrew
>
> Yes, the pkt will look like a normal packet itself.
> In our case HW strips the DSA tag from the packet and forwards it to a VF.

DSA-aware SR-IOV on master, very interesting. I expect that the reverse
is true as well: on xmit, the VF will automatically insert a FWD tag
into the packet too, which encodes a 'virtual' source port and switch id.

What Marvell controller is this? How is the SR-IOV implemented in the
kernel, is it modeled as switchdev, with host representors for the VFs?
Does it support VEPA mode or VEB too? If it supports VEB, does it learn
(more like 'steal') addresses from the DSA switch's FDB? Does it also
push addresses into the DSA switch's FDB?

To your question: why stop at hashing? What about flow steering?
What does your hardware actually support? It is obvious that it has deep
parsing of Marvell DSA tags specifically, so the generic 'masked u32 key'
matching may not be the best way to model this. Can your DSA master even
perform masked matching on generic u32 keys, or are you planning to just
look for the particular pattern of a Marvell DSA tag in the ethtool
rxnfc callbacks, and reject anything that doesn't match?

Re: Query on new ethtool RSS hashing options

[Andrew Lunn]

On Sat, Mar 06, 2021 at 06:04:14PM +0530, Sunil Kovvuri wrote:
> On Sat, Mar 6, 2021 at 5:47 AM Andrew Lunn <andrew@lunn.ch> wrote:
> >
> > On Fri, Mar 05, 2021 at 03:07:02PM -0800, Jakub Kicinski wrote:
> > > On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > > > Hi,
> > > >
> > > > We have a requirement where in we want RSS hashing to be done on packet fields
> > > > which are not currently supported by the ethtool.
> > > >
> > > > Current options:
> > > > ehtool -n <dev> rx-flow-hash
> > > > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> > > >
> > > > Specifically our requirement is to calculate hash with DSA tag (which
> > > > is inserted by switch) plus the TCP/UDP 4-tuple as input.
> > >
> > > Can you share the format of the DSA tag? Is there a driver for it
> > > upstream? Do we need to represent it in union ethtool_flow_union?
> >
> > Sorry, i missed the original question, there was no hint in the
> > subject line that DSA was involved.
> >
> > Why do you want to include DSA tag in the hash? What normally happens
> > with DSA tag drivers is we detect the frame has been received from a
> > switch, and modify where the core flow dissect code looks in the frame
> > to skip over the DSA header and parse the IP header etc as normal.
> 
> I understand your point.
> The requirement to add DSA tag into RSS hashing is coming from one of
> our customer.

So what is the customer requirement? Why does the customer want to do
this? Please explain the real use case.

> > Take a look at net/core/flow_dissect.c:__skb_flow_dissect()
> >
> > This fits with the core ideas of the network stack and offloads. Hide
> > the fact an offload is being used, it should just look like a normal
> > interface.
> >
> >         Andrew
> 
> Yes, the pkt will look like a normal packet itself.
> In our case HW strips the DSA tag from the packet and forwards it to a VF.

So this flow hash will be used only inside the hardware? There is no
software equivalent of it? But the DSA header represents the ingress
port. So why cannot you identify the traffic based on netdev? Isn't
the -n dev equivalent to the DSA header?

    Andrew

Re: Query on new ethtool RSS hashing options

[Tobias Waldekranz]

On Sat, Mar 06, 2021 at 14:54, Vladimir Oltean <olteanv@gmail.com> wrote:
> On Sat, Mar 06, 2021 at 05:38:14PM +0530, Sunil Kovvuri wrote:
>> > Can you share the format of the DSA tag? Is there a driver for it
>> > upstream? Do we need to represent it in union ethtool_flow_union?
>> >
>> 
>> No, there is no driver for this tag in the kernel.
>> I have attached the tag format.
>> There are multiple DSA tag formats and representing them ethtool_flow
>> union would be difficult.
>> Hence wondering if it would be okay to add a more flexible way ie
>> offset and num_bytes from the start of packet.
>
> How sure are you that the tag format you've shared is not identical to
> the one parsed by net/dsa/tag_dsa.c?

That is indeed the format parsed by tag_dsa.c. Based on the layout in
the image, I am pretty sure that it is from the functional spec. of the
Amethyst (6393X). So while the format is supported, that specific device
is not. Hopefully that will change soon:

https://lore.kernel.org/netdev/cover.1610071984.git.pavana.sharma@digi.com/

As for the NIC: Marvell has an EVK for the Amethyst, connected to a
CN9130 SoC. The ethernet controllers in those can parse DSA tags in
hardware, so I would put my money on that.

The upstream driver (mvpp2) does not seem to support it though, AFAIK.

Re: Query on new ethtool RSS hashing options

[Sunil Kovvuri]

On Sat, Mar 6, 2021 at 8:02 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Sat, Mar 06, 2021 at 06:04:14PM +0530, Sunil Kovvuri wrote:
> > On Sat, Mar 6, 2021 at 5:47 AM Andrew Lunn <andrew@lunn.ch> wrote:
> > >
> > > On Fri, Mar 05, 2021 at 03:07:02PM -0800, Jakub Kicinski wrote:
> > > > On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > > > > Hi,
> > > > >
> > > > > We have a requirement where in we want RSS hashing to be done on packet fields
> > > > > which are not currently supported by the ethtool.
> > > > >
> > > > > Current options:
> > > > > ehtool -n <dev> rx-flow-hash
> > > > > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> > > > >
> > > > > Specifically our requirement is to calculate hash with DSA tag (which
> > > > > is inserted by switch) plus the TCP/UDP 4-tuple as input.
> > > >
> > > > Can you share the format of the DSA tag? Is there a driver for it
> > > > upstream? Do we need to represent it in union ethtool_flow_union?
> > >
> > > Sorry, i missed the original question, there was no hint in the
> > > subject line that DSA was involved.
> > >
> > > Why do you want to include DSA tag in the hash? What normally happens
> > > with DSA tag drivers is we detect the frame has been received from a
> > > switch, and modify where the core flow dissect code looks in the frame
> > > to skip over the DSA header and parse the IP header etc as normal.
> >
> > I understand your point.
> > The requirement to add DSA tag into RSS hashing is coming from one of
> > our customer.
>
> So what is the customer requirement? Why does the customer want to do
> this? Please explain the real use case.

Very sorry, after further discussions got to know from customer that
this is not needed.
They were observing some other issue and thought this would solve it.

Thanks,
Sunil.

Re: Query on new ethtool RSS hashing options

[Sunil Kovvuri]

On Sun, Mar 7, 2021 at 2:46 AM Tobias Waldekranz <tobias@waldekranz.com> wrote:
>
> On Sat, Mar 06, 2021 at 14:54, Vladimir Oltean <olteanv@gmail.com> wrote:
> > On Sat, Mar 06, 2021 at 05:38:14PM +0530, Sunil Kovvuri wrote:
> >> > Can you share the format of the DSA tag? Is there a driver for it
> >> > upstream? Do we need to represent it in union ethtool_flow_union?
> >> >
> >>
> >> No, there is no driver for this tag in the kernel.
> >> I have attached the tag format.
> >> There are multiple DSA tag formats and representing them ethtool_flow
> >> union would be difficult.
> >> Hence wondering if it would be okay to add a more flexible way ie
> >> offset and num_bytes from the start of packet.
> >
> > How sure are you that the tag format you've shared is not identical to
> > the one parsed by net/dsa/tag_dsa.c?
>
> That is indeed the format parsed by tag_dsa.c. Based on the layout in
> the image, I am pretty sure that it is from the functional spec. of the
> Amethyst (6393X). So while the format is supported, that specific device
> is not. Hopefully that will change soon:
>
> https://lore.kernel.org/netdev/cover.1610071984.git.pavana.sharma@digi.com/

Thanks for the info.

>
> As for the NIC: Marvell has an EVK for the Amethyst, connected to a
> CN9130 SoC. The ethernet controllers in those can parse DSA tags in
> hardware, so I would put my money on that.
>
> The upstream driver (mvpp2) does not seem to support it though, AFAIK.

This is not for the mvpp2 driver, but for RVU driver
drivers/net/ethernet/marvell/octeontx2.

Thanks,
Sunil.

Re: Query on new ethtool RSS hashing options

[Sunil Kovvuri]

On Sat, Mar 6, 2021 at 6:48 PM Vladimir Oltean <olteanv@gmail.com> wrote:
>
> On Sat, Mar 06, 2021 at 06:04:14PM +0530, Sunil Kovvuri wrote:
> > On Sat, Mar 6, 2021 at 5:47 AM Andrew Lunn <andrew@lunn.ch> wrote:
> > >
> > > On Fri, Mar 05, 2021 at 03:07:02PM -0800, Jakub Kicinski wrote:
> > > > On Fri, 5 Mar 2021 16:15:51 +0530 Sunil Kovvuri wrote:
> > > > > Hi,
> > > > >
> > > > > We have a requirement where in we want RSS hashing to be done on packet fields
> > > > > which are not currently supported by the ethtool.
> > > > >
> > > > > Current options:
> > > > > ehtool -n <dev> rx-flow-hash
> > > > > tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
> > > > >
> > > > > Specifically our requirement is to calculate hash with DSA tag (which
> > > > > is inserted by switch) plus the TCP/UDP 4-tuple as input.
> > > >
> > > > Can you share the format of the DSA tag? Is there a driver for it
> > > > upstream? Do we need to represent it in union ethtool_flow_union?
> > >
> > > Sorry, i missed the original question, there was no hint in the
> > > subject line that DSA was involved.
> > >
> > > Why do you want to include DSA tag in the hash? What normally happens
> > > with DSA tag drivers is we detect the frame has been received from a
> > > switch, and modify where the core flow dissect code looks in the frame
> > > to skip over the DSA header and parse the IP header etc as normal.
> >
> > I understand your point.
> > The requirement to add DSA tag into RSS hashing is coming from one of
> > our customer.
> >
> > >
> > > Take a look at net/core/flow_dissect.c:__skb_flow_dissect()
> > >
> > > This fits with the core ideas of the network stack and offloads. Hide
> > > the fact an offload is being used, it should just look like a normal
> > > interface.
> > >
> > >         Andrew
> >
> > Yes, the pkt will look like a normal packet itself.
> > In our case HW strips the DSA tag from the packet and forwards it to a VF.
>
> DSA-aware SR-IOV on master, very interesting. I expect that the reverse
> is true as well: on xmit, the VF will automatically insert a FWD tag
> into the packet too, which encodes a 'virtual' source port and switch id.

Yes, HW inserts the tag on the transmit side automatically.

>
> What Marvell controller is this? How is the SR-IOV implemented in the
> kernel, is it modeled as switchdev, with host representors for the VFs?

This is Marvell OcteonTx2/CN10K RVU controller
drivers/net/ethernet/marvell/octeontx2.
And no the controller doesn't have a internal switch and hence
currently there is no switchdev support.
The switch I referred to is an external one whose CPU port is
connected to this controller.

> Does it support VEPA mode or VEB too? If it supports VEB, does it learn
> (more like 'steal') addresses from the DSA switch's FDB? Does it also
> push addresses into the DSA switch's FDB?
>
> To your question: why stop at hashing? What about flow steering?
> What does your hardware actually support? It is obvious that it has deep
> parsing of Marvell DSA tags specifically, so the generic 'masked u32 key'
> matching may not be the best way to model this. Can your DSA master even
> perform masked matching on generic u32 keys, or are you planning to just
> look for the particular pattern of a Marvell DSA tag in the ethtool
> rxnfc callbacks, and reject anything that doesn't match?

The parser in the HW is configurable and can parse any header with a mask.

Thanks,
Sunil.

Re: Query on new ethtool RSS hashing options

[Sunil Kovvuri]

On Fri, Mar 5, 2021 at 4:15 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
>
> Hi,
>
> We have a requirement where in we want RSS hashing to be done on packet fields
> which are not currently supported by the ethtool.
>
> Current options:
> ehtool -n <dev> rx-flow-hash
> tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r
>
> Specifically our requirement is to calculate hash with DSA tag (which
> is inserted by switch) plus the TCP/UDP 4-tuple as input.
>
> Is it okay to add such options to the ethtool ?
> or will it be better to add a generic option to take pkt data offset
> and number of bytes ?
>
> Something like
> ethtool -n <dev> rx-flow-hash tcp4 sdfn off <offset in the pkt> num
> <number of bytes/bits>
>
> Any comments, please.
>
> Thanks,
> Sunil.

Apologies for wasting your time.
This hashing support is not needed anymore.

Thanks,
Sunil.

Re: Query on new ethtool RSS hashing options

[Andrew Lunn]

> This is Marvell OcteonTx2/CN10K RVU controller
> drivers/net/ethernet/marvell/octeontx2.
> And no the controller doesn't have a internal switch and hence
> currently there is no switchdev support.
> The switch I referred to is an external one whose CPU port is
> connected to this controller.

Hi Sunil

That is a classic Linux DSA setup. I know it is no longer an issue,
but you might want to look at how switches are supported in Linux. I'm
sure there are people here who are interested in the setup, and having
the MAC do the DSA header work in hardware poses interesting questions
for the Linux DSA architecture. What would a tag driver in
net/dsa/tag_<FOO>.c look like, how do we tell the MAC about the switch
ports it needs to have slave interfaces for, etc.

      Andrew