Re: [PATCH v5 3/8] ethdev: reserve capability flags for PMD-specific API

[Adrien Mazarguil <adrien.mazarguil@6wind.com>]

Hi Konstantin,

On Sun, Jan 08, 2017 at 12:39:55PM +0000, Ananyev, Konstantin wrote:
> 
> Hi Adrien,
> 
> > 
> > Hi Konstantin,
> > 
> > On Thu, Jan 05, 2017 at 11:32:38AM +0000, Ananyev, Konstantin wrote:
> > > Hi Adrien,
[...]
> > > > PMD-specific symbols have nothing to do in the global namespace in my
> > > > opinion, they are not versioned and may evolve without notice. Neither
> > > > applications nor the bonding PMD can rely on them. That's the trade-off.
> > >
> > > Not sure I do understand your reasoning.
> > > For me MACSEC offload is just one of many HW offloads that we support
> > > and should be treated in exactly the same way.
> > > Applications should be able to use it in a transparent and reliable way,
> > > not only under some limited conditions.
> > > Otherwise what is the point to introduce it at all?
> > 
> > Well my first reply to this thread was asking why isn't the whole API global
> > from the start then?
> 
> That's good question, and my preference would always be to have the
> API to configure this feature as generic one.
> I guess the main reason why it is not right now we don't reach an agreement
> how this API should look like: 
> http://dpdk.org/ml/archives/dev/2016-September/047810.html
> But I'll leave it to the author to provide the real reason here. 
> 
> > Given there are valid reasons for it not to and no plan to make it so in the
> > near future, applications must be aware that they are including
> > rte_pmd_ixgbe.h to use it. That in itself is a limiting condition, right?
> 
> Yes, it is definitely a limiting factor.
> Though even if API to configure device to use macsec would be PMD specific right now,
> The API to query that capability and the API to use it at datapath (mbuf.ol_flags) still
> can be (and I think should be) device independent and transparent to use.  

With RESERVED flags, what is global and transparent is the fact the mbuf or
device features in question are PMD-specific and some extra knowledge about
the underlying port is necessary to handle them. I think that could be
useful to applications in order to get the necessary precautions.

> > > Yes, right now it is supported only by ixgbe PMD, but why that should be the
> > > reason to treat is as second-class citizen?
> > > Let say PKT_TX_TUNNEL_* offloads also are supported only by one PMD right now.
> > 
> > You are right about PKT_TX_TUNNEL_*, however these flags exist on their own
> > and are not tied to any API function calls, unlike in this series where
> > PKT_TX_MACSEC can only be used if the DEV_TX_OFFLOAD_MACSEC_INSERT
> > capability is present 
> 
> I don't think PKT_TX_TUNNEL_* 'exists on its own'.
> To use it well behaving app have to:
> 1) Query that device does provide that capability: DEV_TX_OFFLOAD_*_TNL_TSO
> 2) configure PMD( & device) to use that capability
> 3) use that offload at run-time TX code (mb->ol_flags |= ...; mb->tx_offload = ...)
> 
> For PKT_TX_TUNNEL_*  2) is pretty simple - user just need to make sure
> that full-featured TX function will be selected:
> txconf.txq_flags = 0; ...;  rte_eth_tx_queue_setup(..., &txconf);

Pretty much like any other offloads. Any PMD could implement them as well
without exposing additional callbacks.

> For TX_MACSEC, as I understand 2) will be more complicated and
> right now is PMD specific, but anyway the main pattern remains the same.
> So at least 1) and 3) could be kept device neutral.

Yes, however this discussion is precisely because I think it could be a
problem that this API "right now is PMD specific", particularly because it
will remain that way.

> >and the whole thing was configured through
> > rte_pmd_ixgbe_macsec_*() calls after including rte_pmd_ixgbe.h.
> > 
> > To be clear it is not about MACsec per se (as a standardized protocol, I
> > think related definitions for offloads have their place), but it has to do
> > with the fact that the rest of the API is PMD-specific and there is a
> > dependency between them.
> > 
> > > > Therefore until APIs are made global, the safe compromise is to define
> > > > neutral, reserved symbols that any PMD can use to implement their own
> > > > temporary APIs for testing purposes. These can be renamed later without
> > > > changing their value as long as a single PMD uses them.
> > >
> > > Ok, so what we'll gain by introducing PKT_TX_RESERVED instead of PKT_TX_MACSEC?
> > > As I said in my previous mail the redefinition for the same ol_flag bit (and dev capabilities)
> > > by different PMD might create a lot of confusion in future.
> > > Does the potential saving of 1 bit really worth it?
> > 
> > That is one benefit, but my point is mainly to keep applications aware that
> > they are using an API defined by a single PMD, which may be temporary and
> > whose symbols are not versioned.
> 
> As applications have to use PMD specific functions to configure it they definitely are aware.

Therefore they also know if they are only running on top of ixgbe adapters
of a mix with something else. By choosing the PMD-specific path, they know
what they are getting into.

> > Consider this:
> > 
> > rte_mbuf.h:
> > 
> >  #define PKT_TX_RESERVED_0 (1 << 42)
> > 
> > rte_pmd_ixgbe.h:
> > 
> >  #define PKT_TX_MACSEC PKT_TX_RESERVED_0
> > 
> > That way, applications have to get the PKT_TX_MACSEC definition where the
> > rest of the API is also defined.
> > 
> > Other PMDs may reuse PKT_TX_RESERVED_0 and other reserved flags to implement
> > their own experimental APIs.
> 
> That's the main thing I am opposed to.
> I think that by allowing PMD to redefine meaning of
> mbuf.ol_flags and dev_info.(rx| tx)_offload_capa 
> we just asking for trouble.
> 
> Let say tomorrow, i40e will redefine DEV_TX_OFFLOAD_RESERVED_0 and PKT_TX_RESERVED_0
> to implement new specific TX offload (PKT_TX_FEATURE_X).
> Now let say we have an application that works over both ixgbe and i40e
> and would like to use both TX_MACSEC and TC_FEATURE_X offloads whenever they are available.
> As I can see, with the approach you proposed the only way for the application to make it
> is to support 2 different TX code paths (or at least some parts of it).
> To me that way looks inconvenient to the users and source of future troubles.

Remember applications still have to clear PKT_TX_MACSEC and other flags on
ports that do not implement them, just like when the related offload is not
configured even if supported. Whichever approach is taken, applications have
to be careful and need a few extra checks in their TX path. There is no
extra cost involved compared to existing offloads.

> Same for RX:  somewhere at upper layer user got a packet with PKT_RX_RESERVED_0 set.
> What does it really mean if there are different NIC types in the system?

For RX, I agree things are more complicated, applications would have to know
what a given flag means from a particular port. We could define several
RESERVED_X flags that do not overlap on a case basis, so at least
applications know they are dealing with PMD-specific flags.

> > Applications and the bonding PMD can easily be made aware that such reserved
> > flags cannot be shared between ports unless they know what the underlying
> > PMD is, which is already a requirement to use this API in the first place
> > (for instance, calling rte_pmd_ixgbe_macsec_*() functions with another
> > vendor's port_id may crash the application).
> 
> I am talking about that code:
> rte_eth_bond_create(const char *name, uint8_t mode, uint8_t socket_id)
> {
> ...
> /* Take the first dev's offload capabilities */
> internals->rx_offload_capa = dev_info.rx_offload_capa;
> internals->tx_offload_capa = dev_info.tx_offload_capa;
> ...
> internals->rx_offload_capa &= dev_info.rx_offload_capa;
> internals->tx_offload_capa &= dev_info.tx_offload_capa;
> 
> Obviously with what you are suggesting it is not valid any more.
> Bonded device need to support a MASK of all device reserved offloads to exclude
> them from common subset. 
> Any user app(/lib) that does similar thing would also have to be changed.

Why can't they clear RESERVED flags as well? We just have to document the
expected behavior. Even if the bonding PMD does not, applications that do
not use those flags (since they are reserved) should not be impacted.

Because the bonding PMD won't forward PMD-specific API calls, I think it's
safer to clear them anyway.

By the way, how do you handle the case where the bonding PMD exposes the
MACSEC feature and as a result the application tries to configure MACSEC
support on the bonding port_id? How is the resulting crash documented in a
generic fashion?

> > So the idea if/when the API is made global is to rename PKT_TX_RESERVED_0 to
> > PKT_TX_MACSEC and keep its original value.
> > 
> > If other PMDs also implemented PKT_TX_RESERVED_0 in the meantime, it is
> > redefined using a different value. If there is no room left to do so, these
> > PMDs are out of luck I guess, and their specific API is disabled/removed
> > until something gets re-designed.
> > 
> > How about this?
> 
> I still think that we shouldn't allow PMDs to redefine mbuf.olflags and
> dev_info.(rx|tx)_offload_capa. 
> See above for my reasons.

I'm still not comfortable with partially global/specific APIs such as this,
because we assume applications are fully aware of the underlying port in
order to use them, while at the same time the related flags are part of the
global namespace, I find it confusing.

If application developers have no problem with that (so far they haven't
commented on this topic, which means they likely agree to go with anything),
I have no reason left to go against the majority.

-- 
Adrien Mazarguil
6WIND