Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Thursday, May 23, 2019 12:29 AM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; dev@dpdk.org;
> thomas@monjalon.net; stable@dpdk.org
> Subject: Re: [EXT] Re: [dpdk-dev] Re: [PATCH] devtools: skip the symbol
> check when map file under drivers
> > > > > > IMO, The name prefix matters. The rte_* should denote it a
> > > > > > DPDK API and application suppose to use it.
> > > > > >
> > > > > It doesn't, its just a convention.  We have no documentation
> > > > > that indicates what the meaning of an rte_* prefix is
> > > > > specficially, above and beyond the fact thats how we name
> > > > > functions in the DPDK.  If you want to submit a patch to
> > > > > formalize the meaning of function prefixes, you're welcome too
> > > > > (though I won't support it, perhaps others will).  But even if
> > > > > you do, it doesn't address the underlying problem, which is that
> applications still have access to those symbols.
> > > > > Maintaining an ABI by assertion of prefix is really a lousy way
> > > > > to communicate what functions should be accessed by an
> > > > > application and which shouldn't.  If a function is exported, and
> > > > > included in the header file, people will try to use
> > > >
> > > > The current scheme in the driver/common is that, the header files
> > > > are NOT made It as public ie not installed make install.
> > > > The consumer driver includes that using relative path wrt DPDK
> > > > source
> > > directory.
> > > >
> > > Well, thats a step in the right direction.  I'd still like to see
> > > some enforcement to prevent the inadvertent use of those APIs though
> >
> > Yes header file  is  not exported. Not sure how a client can use those.
> > Other than doing some hacking.
> >
> Yes, self prototyping the exported functions would be a way around that.
> > >
> > > > Anyway I will add experimental section to make tool happy.
> > > >
> > > That really not the right solution.  Marking them as experimental is
> > > just papering over the problem, and suggests to users that they will
> > > one day be stable.
> >
> > That what my original concern.
> >
> > > What you want is to explicitly mark those symbols as internal only,
> > > so that any inadvertent use gets flagged.
> >
> > What is your final thought? I can assume the following for my patch
> > generation
> >
> > # No need to mark as experimental
> > # Add @internal to denote it is a internal function like followed some places
> in EAL.
> >
> These are both correct, yes.
> 
> In addition, I would like to see some mechanism that explicitly marks the
> function as exported only for the purposes of internal use.  I understand that
> yours is a case in which this is not expressly needed because you don't
> prototype those functions, but what I'd like to see is a macro in rte_compat.h
> somewhere like this:
> 
> #define INTERNAL_USE_ONLY do {static_assert(0, "Function is only available
> for internal DPDK usage");} while(0)
> 
> so that, in your exported header file (of which I'm sure you have one, even if
> it doesn't contain your private functions, you can do something like this:
> 
> #ifdef BUILDING_RTE_SDK
> void somefunc(int val);
> #else
> #define somefunc(x) INTERNAL_USE_ONLY
> #endif

I think, We have two cases
1) Internal functions are NOT available via  DPDK SDK exported header files
2) Internal functions are available via DPDK SDK exported header files

I think, you are trying to address case 2( as case 1 is not applicable in this context due lack of header file)
For case 2, IMO, the above scheme will not be enough as 
The consumer entity can simply add the exact C flags to skip that check in this case, -DBUILDING_RTE_SDK.
IMO, it would be correct remove private functions from public header files. No strong options on this.
 
> 
> This combination allows for 'internal' functions to be used (defining internal
> to mean access to functions only when building the DPDK SDK), while
> expressly breaking the build of any application which attempts to use these
> functions when not building the SDK (i.e. when building an application that
> expects to link to the DPDK after its built).  Again, I uderstand that in your
> case, it may be sufficient to just not prototype the functions you don't want
> used, but I think in the general case its important to have some mechanism
> to expressly prevent their usage outside the SDK
> 
> Best
> Neil
> 
> > >
> > > Neil
> > > >
> > > >
> >

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Thu, May 23, 2019 at 02:21:29PM +0000, Jerin Jacob Kollanukkaran wrote:
> 
> 
> > -----Original Message-----
> > From: Neil Horman <nhorman@tuxdriver.com>
> > Sent: Thursday, May 23, 2019 12:29 AM
> > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > Cc: Bruce Richardson <bruce.richardson@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net; stable@dpdk.org
> > Subject: Re: [EXT] Re: [dpdk-dev] Re: [PATCH] devtools: skip the symbol
> > check when map file under drivers
> > > > > > > IMO, The name prefix matters. The rte_* should denote it a
> > > > > > > DPDK API and application suppose to use it.
> > > > > > >
> > > > > > It doesn't, its just a convention.  We have no documentation
> > > > > > that indicates what the meaning of an rte_* prefix is
> > > > > > specficially, above and beyond the fact thats how we name
> > > > > > functions in the DPDK.  If you want to submit a patch to
> > > > > > formalize the meaning of function prefixes, you're welcome too
> > > > > > (though I won't support it, perhaps others will).  But even if
> > > > > > you do, it doesn't address the underlying problem, which is that
> > applications still have access to those symbols.
> > > > > > Maintaining an ABI by assertion of prefix is really a lousy way
> > > > > > to communicate what functions should be accessed by an
> > > > > > application and which shouldn't.  If a function is exported, and
> > > > > > included in the header file, people will try to use
> > > > >
> > > > > The current scheme in the driver/common is that, the header files
> > > > > are NOT made It as public ie not installed make install.
> > > > > The consumer driver includes that using relative path wrt DPDK
> > > > > source
> > > > directory.
> > > > >
> > > > Well, thats a step in the right direction.  I'd still like to see
> > > > some enforcement to prevent the inadvertent use of those APIs though
> > >
> > > Yes header file  is  not exported. Not sure how a client can use those.
> > > Other than doing some hacking.
> > >
> > Yes, self prototyping the exported functions would be a way around that.
> > > >
> > > > > Anyway I will add experimental section to make tool happy.
> > > > >
> > > > That really not the right solution.  Marking them as experimental is
> > > > just papering over the problem, and suggests to users that they will
> > > > one day be stable.
> > >
> > > That what my original concern.
> > >
> > > > What you want is to explicitly mark those symbols as internal only,
> > > > so that any inadvertent use gets flagged.
> > >
> > > What is your final thought? I can assume the following for my patch
> > > generation
> > >
> > > # No need to mark as experimental
> > > # Add @internal to denote it is a internal function like followed some places
> > in EAL.
> > >
> > These are both correct, yes.
> > 
> > In addition, I would like to see some mechanism that explicitly marks the
> > function as exported only for the purposes of internal use.  I understand that
> > yours is a case in which this is not expressly needed because you don't
> > prototype those functions, but what I'd like to see is a macro in rte_compat.h
> > somewhere like this:
> > 
> > #define INTERNAL_USE_ONLY do {static_assert(0, "Function is only available
> > for internal DPDK usage");} while(0)
> > 
> > so that, in your exported header file (of which I'm sure you have one, even if
> > it doesn't contain your private functions, you can do something like this:
> > 
> > #ifdef BUILDING_RTE_SDK
> > void somefunc(int val);
> > #else
> > #define somefunc(x) INTERNAL_USE_ONLY
> > #endif
> 
> I think, We have two cases
> 1) Internal functions are NOT available via  DPDK SDK exported header files
> 2) Internal functions are available via DPDK SDK exported header files
> 
> I think, you are trying to address case 2( as case 1 is not applicable in this context due lack of header file)
> For case 2, IMO, the above scheme will not be enough as 
> The consumer entity can simply add the exact C flags to skip that check in this case, -DBUILDING_RTE_SDK.
> IMO, it would be correct remove private functions from public header files. No strong options on this.
>  

I'm thinking about it a bit differently.  Internal functions should never be
available to user, weather they are prototyped in DPDK header files or not.
Unfortunately, because of how library symbol exports work, there is no way to
differentiate between which exported functions are internal or external, they
are only exported or not, and as such, they are always resolveable by someone
linking against them (regardless of which hackery is used to achieve that
result).  I'd like a way to prevent users who are only using the SDK (not
building it) from accessing those symbols, and the above is the best solution I
can come up with.  I admit its not great, but it does place a roadblock in the
way of users who attempt to use symbols we don't want to give them access to.
And yes its circumventable by defining BUILDING_RTE_SDK, but I would think its
clear that they are not building the SDK, and so they should not be doing that.

Just not exporting the requisite header files is an easier solution, so if thats
the consensus I can be ok with that, but I would really love to have a way to
document in the code those functions which are not meant for external
consumption.

Neil

> > 
> > This combination allows for 'internal' functions to be used (defining internal
> > to mean access to functions only when building the DPDK SDK), while
> > expressly breaking the build of any application which attempts to use these
> > functions when not building the SDK (i.e. when building an application that
> > expects to link to the DPDK after its built).  Again, I uderstand that in your
> > case, it may be sufficient to just not prototype the functions you don't want
> > used, but I think in the general case its important to have some mechanism
> > to expressly prevent their usage outside the SDK
> > 
> > Best
> > Neil
> > 
> > > >
> > > > Neil
> > > > >
> > > > >
> > >
> 

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Thomas Monjalon]

23/05/2019 19:57, Neil Horman:
> On Thu, May 23, 2019 at 02:21:29PM +0000, Jerin Jacob Kollanukkaran wrote:
> > From: Neil Horman <nhorman@tuxdriver.com>
> > > > > > > > IMO, The name prefix matters. The rte_* should denote it a
> > > > > > > > DPDK API and application suppose to use it.
> > > > > > > >
> > > > > > > It doesn't, its just a convention.  We have no documentation
> > > > > > > that indicates what the meaning of an rte_* prefix is
> > > > > > > specficially, above and beyond the fact thats how we name
> > > > > > > functions in the DPDK.  If you want to submit a patch to
> > > > > > > formalize the meaning of function prefixes, you're welcome too
> > > > > > > (though I won't support it, perhaps others will).  But even if
> > > > > > > you do, it doesn't address the underlying problem, which is that
> > > applications still have access to those symbols.
> > > > > > > Maintaining an ABI by assertion of prefix is really a lousy way
> > > > > > > to communicate what functions should be accessed by an
> > > > > > > application and which shouldn't.  If a function is exported, and
> > > > > > > included in the header file, people will try to use
> > > > > >
> > > > > > The current scheme in the driver/common is that, the header files
> > > > > > are NOT made It as public ie not installed make install.
> > > > > > The consumer driver includes that using relative path wrt DPDK
> > > > > > source
> > > > > directory.
> > > > > >
> > > > > Well, thats a step in the right direction.  I'd still like to see
> > > > > some enforcement to prevent the inadvertent use of those APIs though
> > > >
> > > > Yes header file  is  not exported. Not sure how a client can use those.
> > > > Other than doing some hacking.
> > > >
> > > Yes, self prototyping the exported functions would be a way around that.
> > > > >
> > > > > > Anyway I will add experimental section to make tool happy.
> > > > > >
> > > > > That really not the right solution.  Marking them as experimental is
> > > > > just papering over the problem, and suggests to users that they will
> > > > > one day be stable.
> > > >
> > > > That what my original concern.
> > > >
> > > > > What you want is to explicitly mark those symbols as internal only,
> > > > > so that any inadvertent use gets flagged.
> > > >
> > > > What is your final thought? I can assume the following for my patch
> > > > generation
> > > >
> > > > # No need to mark as experimental
> > > > # Add @internal to denote it is a internal function like followed some places
> > > in EAL.
> > > >
> > > These are both correct, yes.
> > > 
> > > In addition, I would like to see some mechanism that explicitly marks the
> > > function as exported only for the purposes of internal use.  I understand that
> > > yours is a case in which this is not expressly needed because you don't
> > > prototype those functions, but what I'd like to see is a macro in rte_compat.h
> > > somewhere like this:
> > > 
> > > #define INTERNAL_USE_ONLY do {static_assert(0, "Function is only available
> > > for internal DPDK usage");} while(0)
> > > 
> > > so that, in your exported header file (of which I'm sure you have one, even if
> > > it doesn't contain your private functions, you can do something like this:
> > > 
> > > #ifdef BUILDING_RTE_SDK
> > > void somefunc(int val);
> > > #else
> > > #define somefunc(x) INTERNAL_USE_ONLY
> > > #endif
> > 
> > I think, We have two cases
> > 1) Internal functions are NOT available via  DPDK SDK exported header files
> > 2) Internal functions are available via DPDK SDK exported header files
> > 
> > I think, you are trying to address case 2( as case 1 is not applicable in this context due lack of header file)
> > For case 2, IMO, the above scheme will not be enough as 
> > The consumer entity can simply add the exact C flags to skip that check in this case, -DBUILDING_RTE_SDK.
> > IMO, it would be correct remove private functions from public header files. No strong options on this.
> >  
> 
> I'm thinking about it a bit differently.  Internal functions should never be
> available to user, weather they are prototyped in DPDK header files or not.
> Unfortunately, because of how library symbol exports work, there is no way to
> differentiate between which exported functions are internal or external, they
> are only exported or not, and as such, they are always resolveable by someone
> linking against them (regardless of which hackery is used to achieve that
> result).  I'd like a way to prevent users who are only using the SDK (not
> building it) from accessing those symbols, and the above is the best solution I
> can come up with.  I admit its not great, but it does place a roadblock in the
> way of users who attempt to use symbols we don't want to give them access to.
> And yes its circumventable by defining BUILDING_RTE_SDK, but I would think its
> clear that they are not building the SDK, and so they should not be doing that.
> 
> Just not exporting the requisite header files is an easier solution, so if thats
> the consensus I can be ok with that, but I would really love to have a way to
> document in the code those functions which are not meant for external
> consumption.

I think there are good ideas here.
Please come with a patch and we'll try to apply the chosen policy
to the existing code.

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Thu, May 23, 2019 at 08:59:07PM +0200, Thomas Monjalon wrote:
> 23/05/2019 19:57, Neil Horman:
> > On Thu, May 23, 2019 at 02:21:29PM +0000, Jerin Jacob Kollanukkaran wrote:
> > > From: Neil Horman <nhorman@tuxdriver.com>
> > > > > > > > > IMO, The name prefix matters. The rte_* should denote it a
> > > > > > > > > DPDK API and application suppose to use it.
> > > > > > > > >
> > > > > > > > It doesn't, its just a convention.  We have no documentation
> > > > > > > > that indicates what the meaning of an rte_* prefix is
> > > > > > > > specficially, above and beyond the fact thats how we name
> > > > > > > > functions in the DPDK.  If you want to submit a patch to
> > > > > > > > formalize the meaning of function prefixes, you're welcome too
> > > > > > > > (though I won't support it, perhaps others will).  But even if
> > > > > > > > you do, it doesn't address the underlying problem, which is that
> > > > applications still have access to those symbols.
> > > > > > > > Maintaining an ABI by assertion of prefix is really a lousy way
> > > > > > > > to communicate what functions should be accessed by an
> > > > > > > > application and which shouldn't.  If a function is exported, and
> > > > > > > > included in the header file, people will try to use
> > > > > > >
> > > > > > > The current scheme in the driver/common is that, the header files
> > > > > > > are NOT made It as public ie not installed make install.
> > > > > > > The consumer driver includes that using relative path wrt DPDK
> > > > > > > source
> > > > > > directory.
> > > > > > >
> > > > > > Well, thats a step in the right direction.  I'd still like to see
> > > > > > some enforcement to prevent the inadvertent use of those APIs though
> > > > >
> > > > > Yes header file  is  not exported. Not sure how a client can use those.
> > > > > Other than doing some hacking.
> > > > >
> > > > Yes, self prototyping the exported functions would be a way around that.
> > > > > >
> > > > > > > Anyway I will add experimental section to make tool happy.
> > > > > > >
> > > > > > That really not the right solution.  Marking them as experimental is
> > > > > > just papering over the problem, and suggests to users that they will
> > > > > > one day be stable.
> > > > >
> > > > > That what my original concern.
> > > > >
> > > > > > What you want is to explicitly mark those symbols as internal only,
> > > > > > so that any inadvertent use gets flagged.
> > > > >
> > > > > What is your final thought? I can assume the following for my patch
> > > > > generation
> > > > >
> > > > > # No need to mark as experimental
> > > > > # Add @internal to denote it is a internal function like followed some places
> > > > in EAL.
> > > > >
> > > > These are both correct, yes.
> > > > 
> > > > In addition, I would like to see some mechanism that explicitly marks the
> > > > function as exported only for the purposes of internal use.  I understand that
> > > > yours is a case in which this is not expressly needed because you don't
> > > > prototype those functions, but what I'd like to see is a macro in rte_compat.h
> > > > somewhere like this:
> > > > 
> > > > #define INTERNAL_USE_ONLY do {static_assert(0, "Function is only available
> > > > for internal DPDK usage");} while(0)
> > > > 
> > > > so that, in your exported header file (of which I'm sure you have one, even if
> > > > it doesn't contain your private functions, you can do something like this:
> > > > 
> > > > #ifdef BUILDING_RTE_SDK
> > > > void somefunc(int val);
> > > > #else
> > > > #define somefunc(x) INTERNAL_USE_ONLY
> > > > #endif
> > > 
> > > I think, We have two cases
> > > 1) Internal functions are NOT available via  DPDK SDK exported header files
> > > 2) Internal functions are available via DPDK SDK exported header files
> > > 
> > > I think, you are trying to address case 2( as case 1 is not applicable in this context due lack of header file)
> > > For case 2, IMO, the above scheme will not be enough as 
> > > The consumer entity can simply add the exact C flags to skip that check in this case, -DBUILDING_RTE_SDK.
> > > IMO, it would be correct remove private functions from public header files. No strong options on this.
> > >  
> > 
> > I'm thinking about it a bit differently.  Internal functions should never be
> > available to user, weather they are prototyped in DPDK header files or not.
> > Unfortunately, because of how library symbol exports work, there is no way to
> > differentiate between which exported functions are internal or external, they
> > are only exported or not, and as such, they are always resolveable by someone
> > linking against them (regardless of which hackery is used to achieve that
> > result).  I'd like a way to prevent users who are only using the SDK (not
> > building it) from accessing those symbols, and the above is the best solution I
> > can come up with.  I admit its not great, but it does place a roadblock in the
> > way of users who attempt to use symbols we don't want to give them access to.
> > And yes its circumventable by defining BUILDING_RTE_SDK, but I would think its
> > clear that they are not building the SDK, and so they should not be doing that.
> > 
> > Just not exporting the requisite header files is an easier solution, so if thats
> > the consensus I can be ok with that, but I would really love to have a way to
> > document in the code those functions which are not meant for external
> > consumption.
> 
> I think there are good ideas here.
> Please come with a patch and we'll try to apply the chosen policy
> to the existing code.
> 
Ok, I'll give it a shot
Neil

> 
> 

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Wednesday, May 22, 2019 6:43 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; dev@dpdk.org;
> thomas@monjalon.net; stable@dpdk.org
> Subject: [EXT] Re: [dpdk-dev] Re: [PATCH] devtools: skip the symbol check
> when map file under drivers
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Wed, May 22, 2019 at 11:54:13AM +0000, Jerin Jacob Kollanukkaran
> wrote:
> > > -----Original Message-----
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > Sent: Wednesday, May 22, 2019 4:21 PM
> > > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > Cc: Neil Horman <nhorman@tuxdriver.com>; dev@dpdk.org;
> > > thomas@monjalon.net; stable@dpdk.org
> > > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH] devtools: skip the symbol
> > > check when map file under drivers
> > >
> > > > > Sorry, but I'm not ok with this, because many of our DPDK PMDs
> > > > > have functions that get exported which are meant to be called by
> > > > > applications directly.  The
> > > >
> > > > OK. Just to update my knowledge, Should those API needs to go
> > > > through ABI/API depreciation process?
> > > >
> > > > Actually, I am concerned about the APIs, which is called between
> > > > drviers not the application. For example,
> > > > drivers/common/dpaax/rte_common_dpaax_version.map
> > > >
> > > > it is not interface to application rather it is for intra driver case.
> > > > I think, I can change my logic to Skip the symbols which NOT
> > > > starting with
> > > rte_.
> > > > Agree?
> > > >
> > > > Context:
> > > > I am adding a new driver/common/octeontx2 directory and it has
> > > > some API which Needs to shared between drivers not to the
> > > > application. For me, it does not make sense to go through any ABI
> process in such case.
> > > >
> > > >
> > > Maybe not, but other drivers will have APIs designed for apps to
> > > call directly - some NIC drivers have them, and I suspect that
> > > rawdev drivers will need them a lot. Therefore, it's best to have
> > > the drivers directory scanned by our tooling.
> >
> > Agreed. But all of those API  which called directly called from
> > application is starts with rte_ symbol. How about skipping the symbols
> > which is NOT start with rte_*
> > example:
> > drivers/common/octeontx/rte_common_octeontx_version.map
> > drivers/common/dpaax/rte_common_dpaax_version.map
> >
> 
> No, that won't work.  If you export a function, it doesn't matter if its named
> rte_* or not.  Its accessible from any library/application that cares to call it,

IMO, The name prefix matters. The rte_* should denote it a DPDK API and application
suppose to use it.

I don't think, giving experimental status to intra driver API helps anyone, neither driver nor
application.

If you think strongly that experimental needs to be added for intra driver APIs then I can add that.
 

> and so you have a responsibility to keep it stable for those users.
> 
> Currently the way we have around that is the use of the __rte_experimental
> tag.
> Adding that tag to an exported function marks it as being unstable, and while
> you can use it, it will generate a build time warning about its use, unless you
> define ALLOW_EXPERIMENTAL_API.  You could use that, understanding that
> in-tree drivers could use it safely, as you should always be keeping the API in
> sync with its users, but thats not quite what you want I don't think.
> 
> Another solution (allbeit a slightly risky one), would be to bifurcate your
> header files into a public and private version, with the private version
> prototyping your driver-only functions properly, and the public version
> aliasing them such that they generate a build time error indicating those
> functions aren't available for public use (you can use the gcc static_assert
> macro I believe).  Users could circumvent it by pulling the private header out
> of the build, or just prototyping the functions themselves, but at that point a
> user is asking for trouble anyway
> 
> Neil

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Wed, May 22, 2019 at 01:41:03PM +0000, Jerin Jacob Kollanukkaran wrote:
> > -----Original Message-----
> > From: Neil Horman <nhorman@tuxdriver.com>
> > Sent: Wednesday, May 22, 2019 6:43 PM
> > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > Cc: Bruce Richardson <bruce.richardson@intel.com>; dev@dpdk.org;
> > thomas@monjalon.net; stable@dpdk.org
> > Subject: [EXT] Re: [dpdk-dev] Re: [PATCH] devtools: skip the symbol check
> > when map file under drivers
> > 
> > External Email
> > 
> > ----------------------------------------------------------------------
> > On Wed, May 22, 2019 at 11:54:13AM +0000, Jerin Jacob Kollanukkaran
> > wrote:
> > > > -----Original Message-----
> > > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > Sent: Wednesday, May 22, 2019 4:21 PM
> > > > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > > Cc: Neil Horman <nhorman@tuxdriver.com>; dev@dpdk.org;
> > > > thomas@monjalon.net; stable@dpdk.org
> > > > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH] devtools: skip the symbol
> > > > check when map file under drivers
> > > >
> > > > > > Sorry, but I'm not ok with this, because many of our DPDK PMDs
> > > > > > have functions that get exported which are meant to be called by
> > > > > > applications directly.  The
> > > > >
> > > > > OK. Just to update my knowledge, Should those API needs to go
> > > > > through ABI/API depreciation process?
> > > > >
> > > > > Actually, I am concerned about the APIs, which is called between
> > > > > drviers not the application. For example,
> > > > > drivers/common/dpaax/rte_common_dpaax_version.map
> > > > >
> > > > > it is not interface to application rather it is for intra driver case.
> > > > > I think, I can change my logic to Skip the symbols which NOT
> > > > > starting with
> > > > rte_.
> > > > > Agree?
> > > > >
> > > > > Context:
> > > > > I am adding a new driver/common/octeontx2 directory and it has
> > > > > some API which Needs to shared between drivers not to the
> > > > > application. For me, it does not make sense to go through any ABI
> > process in such case.
> > > > >
> > > > >
> > > > Maybe not, but other drivers will have APIs designed for apps to
> > > > call directly - some NIC drivers have them, and I suspect that
> > > > rawdev drivers will need them a lot. Therefore, it's best to have
> > > > the drivers directory scanned by our tooling.
> > >
> > > Agreed. But all of those API  which called directly called from
> > > application is starts with rte_ symbol. How about skipping the symbols
> > > which is NOT start with rte_*
> > > example:
> > > drivers/common/octeontx/rte_common_octeontx_version.map
> > > drivers/common/dpaax/rte_common_dpaax_version.map
> > >
> > 
> > No, that won't work.  If you export a function, it doesn't matter if its named
> > rte_* or not.  Its accessible from any library/application that cares to call it,
> 
> IMO, The name prefix matters. The rte_* should denote it a DPDK API and application
> suppose to use it.
> 
It doesn't, its just a convention.  We have no documentation that indicates what
the meaning of an rte_* prefix is specficially, above and beyond the fact thats
how we name functions in the DPDK.  If you want to submit a patch to formalize
the meaning of function prefixes, you're welcome too (though I won't support it,
perhaps others will).  But even if you do, it doesn't address the underlying
problem, which is that applications still have access to those symbols.
Maintaining an ABI by assertion of prefix is really a lousy way to communicate
what functions should be accessed by an application and which shouldn't.  If a
function is exported, and included in the header file, people will try to use
them.  You need an enforcement mechanism to call attention to the fact that they
are unusable in normal operation

> I don't think, giving experimental status to intra driver API helps anyone, neither driver nor
> application.
> 
Nor do I, I was just using it as an example of how you might create an
enforcement mechanism.  I think a better solution is bifurcating your headers
to a public and private version, in which the former defines apis you wish to be
generally inaccessable to a static_assert that causes a build time error for
uses that are not 'DPDK internal'

> If you think strongly that experimental needs to be added for intra driver APIs then I can add that.
>  
No, I don't, I was just using that as an example of what you might do.  I think
the header solution is far more appropriate for this case.

Neil

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Wednesday, May 22, 2019 6:16 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Cc: dev@dpdk.org; thomas@monjalon.net; stable@dpdk.org
> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH] devtools: skip the symbol check
> when map file under drivers
> 
> On Wed, May 22, 2019 at 03:05:54AM +0000, Jerin Jacob Kollanukkaran
> wrote:
> > > -----Original Message-----
> > > From: Neil Horman <nhorman@tuxdriver.com>
> > > Sent: Wednesday, May 22, 2019 1:57 AM
> > > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > Cc: dev@dpdk.org; thomas@monjalon.net; stable@dpdk.org
> > > Subject: [EXT] Re: [dpdk-dev] [PATCH] devtools: skip the symbol
> > > check when map file under drivers
> > >
> > > On Wed, May 22, 2019 at 01:26:28AM +0530, jerinj@marvell.com wrote:
> > > > From: Jerin Jacob <jerinj@marvell.com>
> > > >
> > > > Drivers do not have ABI.
> > > > Skip the symbol check if map file under drivers directory.
> > > >
> > > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol
> > > > addition")
> > > >
> > > > Cc: stable@dpdk.org
> > > > Cc: Neil Horman <nhorman@tuxdriver.com>
> > > >
> > > Sorry, but I'm not ok with this, because many of our DPDK PMDs have
> > > functions that get exported which are meant to be called by
> > > applications directly.  The
> >
> > OK. Just to update my knowledge, Should those API needs to go through
> > ABI/API depreciation process?
> >
> Yes, they definately should, they are API's just as any other in the core DPDK
> library.

OK


> 
> > Actually, I am concerned about the APIs, which is called between
> > drviers not the application. For example,
> > drivers/common/dpaax/rte_common_dpaax_version.map
> >
> > it is not interface to application rather it is for intra driver case.
> > I think, I can change my logic to Skip the symbols which NOT starting with
> rte_.
> > Agree?
> >
> No, Thats just one case, and if those calls are between drivers, so be it, but
> those still need to be stable, and we have other examples (like the bonding
> or dummy driver), which have additional APIs that are explicitly meant to be
> used by an application.

There is no disagreement on the API that exposed to application.
I am concerned with internal driver APIs. For example, I am getting following warning

ERROR: symbol otx2_mbox_alloc_msg_rsp is added in the DPDK_19.05 section, but is expected to be added in the EXPERIMENTAL section of the version map

This API suppose to be called only a octeontx2 network driver from octeontx2 common driver
i.e application should not expect any stability on intra driver functions or it does not meant to
be used by application.

Thomas,
Any thought on this?



> 
> > Context:
> > I am adding a new driver/common/octeontx2 directory and it has some
> > API which Needs to shared between drivers not to the application. For
> > me, it does not make sense to go through any ABI process in such case.
> >
> Why?  If you create an API thats reachable from another block of code (be it
> a driver or an application), you've created a dependency in which that API
> must remain stable, lest you risk breaking something.  If an end user writes
> an out-of-tree PMD which makes use of an an additional driver API, then you
> need to keep it stable or you will break them.
> 
> >
> > > dpaa2 driver is a good example, the cryptodev scheduler is another.
> > > Take a look at their version.map files to see what I mean.
> > >
> > > Unless we are willing to make drivers opaque objects that are only
> > > accessible from the [eth|crypto|etc]dev apis in the DPDK core, we
> > > have the potential for exported symbols, which means we have an ABI
> > > that has to be maintained, or at least recognized and reviewed for
> > > consistency
> > >
> > > Nacked-by: Neil Horman <nhorman@tuxdriver.com>
> > >
> > >
> > > > Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> > > > ---
> > > >  devtools/check-symbol-change.sh | 8 ++++++++
> > > >  1 file changed, 8 insertions(+)
> > > >
> > > > diff --git a/devtools/check-symbol-change.sh
> > > > b/devtools/check-symbol-change.sh index c5434f3bb..444beddad
> > > > 100755
> > > > --- a/devtools/check-symbol-change.sh
> > > > +++ b/devtools/check-symbol-change.sh
> > > > @@ -93,6 +93,14 @@ check_for_rule_violations()
> > > >  		if [ "$ar" = "add" ]
> > > >  		then
> > > >
> > > > +			directory=`echo $mname | cut -d / -f 2`
> > > > +			if [ "$directory" = "drivers" ]
> > > > +			then
> > > > +				# Drivers do not have ABI. Skip further
> > > > +				# processing if the map file is under
> > > > +				# drivers directory
> > > > +				continue
> > > > +			fi
> > > >  			if [ "$secname" = "unknown" ]
> > > >  			then
> > > >  				# Just inform the user of this occurrence, but
> > > > --
> > > > 2.21.0
> > > >
> > > >
> >

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Wed, May 22, 2019 at 01:12:34PM +0000, Jerin Jacob Kollanukkaran wrote:
> > -----Original Message-----
> > From: Neil Horman <nhorman@tuxdriver.com>
> > Sent: Wednesday, May 22, 2019 6:16 PM
> > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > Cc: dev@dpdk.org; thomas@monjalon.net; stable@dpdk.org
> > Subject: Re: [EXT] Re: [dpdk-dev] [PATCH] devtools: skip the symbol check
> > when map file under drivers
> > 
> > On Wed, May 22, 2019 at 03:05:54AM +0000, Jerin Jacob Kollanukkaran
> > wrote:
> > > > -----Original Message-----
> > > > From: Neil Horman <nhorman@tuxdriver.com>
> > > > Sent: Wednesday, May 22, 2019 1:57 AM
> > > > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > > Cc: dev@dpdk.org; thomas@monjalon.net; stable@dpdk.org
> > > > Subject: [EXT] Re: [dpdk-dev] [PATCH] devtools: skip the symbol
> > > > check when map file under drivers
> > > >
> > > > On Wed, May 22, 2019 at 01:26:28AM +0530, jerinj@marvell.com wrote:
> > > > > From: Jerin Jacob <jerinj@marvell.com>
> > > > >
> > > > > Drivers do not have ABI.
> > > > > Skip the symbol check if map file under drivers directory.
> > > > >
> > > > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol
> > > > > addition")
> > > > >
> > > > > Cc: stable@dpdk.org
> > > > > Cc: Neil Horman <nhorman@tuxdriver.com>
> > > > >
> > > > Sorry, but I'm not ok with this, because many of our DPDK PMDs have
> > > > functions that get exported which are meant to be called by
> > > > applications directly.  The
> > >
> > > OK. Just to update my knowledge, Should those API needs to go through
> > > ABI/API depreciation process?
> > >
> > Yes, they definately should, they are API's just as any other in the core DPDK
> > library.
> 
> OK
> 
> 
> > 
> > > Actually, I am concerned about the APIs, which is called between
> > > drviers not the application. For example,
> > > drivers/common/dpaax/rte_common_dpaax_version.map
> > >
> > > it is not interface to application rather it is for intra driver case.
> > > I think, I can change my logic to Skip the symbols which NOT starting with
> > rte_.
> > > Agree?
> > >
> > No, Thats just one case, and if those calls are between drivers, so be it, but
> > those still need to be stable, and we have other examples (like the bonding
> > or dummy driver), which have additional APIs that are explicitly meant to be
> > used by an application.
> 
> There is no disagreement on the API that exposed to application.
> I am concerned with internal driver APIs. For example, I am getting following warning
> 
> ERROR: symbol otx2_mbox_alloc_msg_rsp is added in the DPDK_19.05 section, but is expected to be added in the EXPERIMENTAL section of the version map
> 
Thats a warning about the fact that you added an API call in a versioned section
of a library instead of the Experimental section, thats part of our policy.  New
APIs need to go through the experimental tag first.

I understand what you are saying about driver only apis, and I mentioned that in
my other email farther down the thread.  The problem is that "driver only apis"
are currently just a conceptual thing for us to discuss.  They're still,
practially speaking, API's that any downstream user can access and become
dependent on, which we need to manage, either by keeping the API stable, so it
stays usable for all callers, or by developing a way to mark driver only API's
as such.  I proposed a method that you might use to do the latter in my other email.

> This API suppose to be called only a octeontx2 network driver from octeontx2 common driver
> i.e application should not expect any stability on intra driver functions or it does not meant to
> be used by application.
> 
Ok, but again, your assertion is that its driver to driver only, but in
practicaility, that assertion is irrelevant.  Those symbols are still exposed
for general use, and weather or not you say they aren't part of the ABI, the
fact of the matter is, there is no way to tell the difference from a linked
object standpoint.  Instead of hobbling the tool to just not scan anything, you
need to find a way to differentiate these symbols, so that you can enforce your
assertion that there are restrictions on where these APIs are called from.

Neil

> Thomas,
> Any thought on this?
> 
> 
> 
> > 
> > >

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Thomas Monjalon]

22/05/2019 15:40, Neil Horman:
> On Wed, May 22, 2019 at 01:12:34PM +0000, Jerin Jacob Kollanukkaran wrote:
> > From: Neil Horman <nhorman@tuxdriver.com>
> > > On Wed, May 22, 2019 at 03:05:54AM +0000, Jerin Jacob Kollanukkaran
> > > wrote:
> > > > From: Neil Horman <nhorman@tuxdriver.com>
> > > > > On Wed, May 22, 2019 at 01:26:28AM +0530, jerinj@marvell.com wrote:
> > > > > > From: Jerin Jacob <jerinj@marvell.com>
> > > > > >
> > > > > > Drivers do not have ABI.
> > > > > > Skip the symbol check if map file under drivers directory.
> > > > > >
> > > > > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol
> > > > > > addition")
> > > > > >
> > > > > > Cc: stable@dpdk.org
> > > > > > Cc: Neil Horman <nhorman@tuxdriver.com>
> > > > > >
> > > > > Sorry, but I'm not ok with this, because many of our DPDK PMDs have
> > > > > functions that get exported which are meant to be called by
> > > > > applications directly.  The
> > > >
> > > > OK. Just to update my knowledge, Should those API needs to go through
> > > > ABI/API depreciation process?
> > > >
> > > Yes, they definately should, they are API's just as any other in the core DPDK
> > > library.
> > 
> > OK
> > 
> > 
> > > 
> > > > Actually, I am concerned about the APIs, which is called between
> > > > drviers not the application. For example,
> > > > drivers/common/dpaax/rte_common_dpaax_version.map
> > > >
> > > > it is not interface to application rather it is for intra driver case.
> > > > I think, I can change my logic to Skip the symbols which NOT starting with
> > > rte_.
> > > > Agree?
> > > >
> > > No, Thats just one case, and if those calls are between drivers, so be it, but
> > > those still need to be stable, and we have other examples (like the bonding
> > > or dummy driver), which have additional APIs that are explicitly meant to be
> > > used by an application.
> > 
> > There is no disagreement on the API that exposed to application.
> > I am concerned with internal driver APIs. For example, I am getting following warning
> > 
> > ERROR: symbol otx2_mbox_alloc_msg_rsp is added in the DPDK_19.05 section, but is expected to be added in the EXPERIMENTAL section of the version map
> > 
> Thats a warning about the fact that you added an API call in a versioned section
> of a library instead of the Experimental section, thats part of our policy.  New
> APIs need to go through the experimental tag first.
> 
> I understand what you are saying about driver only apis, and I mentioned that in
> my other email farther down the thread.  The problem is that "driver only apis"
> are currently just a conceptual thing for us to discuss.  They're still,
> practially speaking, API's that any downstream user can access and become
> dependent on, which we need to manage, either by keeping the API stable, so it
> stays usable for all callers, or by developing a way to mark driver only API's
> as such.  I proposed a method that you might use to do the latter in my other email.
> 
> > This API suppose to be called only a octeontx2 network driver from octeontx2 common driver
> > i.e application should not expect any stability on intra driver functions or it does not meant to
> > be used by application.
> > 
> Ok, but again, your assertion is that its driver to driver only, but in
> practicaility, that assertion is irrelevant.  Those symbols are still exposed
> for general use, and weather or not you say they aren't part of the ABI, the
> fact of the matter is, there is no way to tell the difference from a linked
> object standpoint.  Instead of hobbling the tool to just not scan anything, you
> need to find a way to differentiate these symbols, so that you can enforce your
> assertion that there are restrictions on where these APIs are called from.
> 
> Neil
> 
> > Thomas,
> > Any thought on this?

As Neil said, we need to differentiate the internal APIs.
We already have this issue in a number of places like EAL, or ethdev,
and it was poorly addressed with some comments like "@internal".

Practically I don't care about stability of these internal functions,
but I agree that it creates a mess in the tooling and confuse users.

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Wed, May 22, 2019 at 04:12:40PM +0200, Thomas Monjalon wrote:
> 22/05/2019 15:40, Neil Horman:
> > On Wed, May 22, 2019 at 01:12:34PM +0000, Jerin Jacob Kollanukkaran wrote:
> > > From: Neil Horman <nhorman@tuxdriver.com>
> > > > On Wed, May 22, 2019 at 03:05:54AM +0000, Jerin Jacob Kollanukkaran
> > > > wrote:
> > > > > From: Neil Horman <nhorman@tuxdriver.com>
> > > > > > On Wed, May 22, 2019 at 01:26:28AM +0530, jerinj@marvell.com wrote:
> > > > > > > From: Jerin Jacob <jerinj@marvell.com>
> > > > > > >
> > > > > > > Drivers do not have ABI.
> > > > > > > Skip the symbol check if map file under drivers directory.
> > > > > > >
> > > > > > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol
> > > > > > > addition")
> > > > > > >
> > > > > > > Cc: stable@dpdk.org
> > > > > > > Cc: Neil Horman <nhorman@tuxdriver.com>
> > > > > > >
> > > > > > Sorry, but I'm not ok with this, because many of our DPDK PMDs have
> > > > > > functions that get exported which are meant to be called by
> > > > > > applications directly.  The
> > > > >
> > > > > OK. Just to update my knowledge, Should those API needs to go through
> > > > > ABI/API depreciation process?
> > > > >
> > > > Yes, they definately should, they are API's just as any other in the core DPDK
> > > > library.
> > > 
> > > OK
> > > 
> > > 
> > > > 
> > > > > Actually, I am concerned about the APIs, which is called between
> > > > > drviers not the application. For example,
> > > > > drivers/common/dpaax/rte_common_dpaax_version.map
> > > > >
> > > > > it is not interface to application rather it is for intra driver case.
> > > > > I think, I can change my logic to Skip the symbols which NOT starting with
> > > > rte_.
> > > > > Agree?
> > > > >
> > > > No, Thats just one case, and if those calls are between drivers, so be it, but
> > > > those still need to be stable, and we have other examples (like the bonding
> > > > or dummy driver), which have additional APIs that are explicitly meant to be
> > > > used by an application.
> > > 
> > > There is no disagreement on the API that exposed to application.
> > > I am concerned with internal driver APIs. For example, I am getting following warning
> > > 
> > > ERROR: symbol otx2_mbox_alloc_msg_rsp is added in the DPDK_19.05 section, but is expected to be added in the EXPERIMENTAL section of the version map
> > > 
> > Thats a warning about the fact that you added an API call in a versioned section
> > of a library instead of the Experimental section, thats part of our policy.  New
> > APIs need to go through the experimental tag first.
> > 
> > I understand what you are saying about driver only apis, and I mentioned that in
> > my other email farther down the thread.  The problem is that "driver only apis"
> > are currently just a conceptual thing for us to discuss.  They're still,
> > practially speaking, API's that any downstream user can access and become
> > dependent on, which we need to manage, either by keeping the API stable, so it
> > stays usable for all callers, or by developing a way to mark driver only API's
> > as such.  I proposed a method that you might use to do the latter in my other email.
> > 
> > > This API suppose to be called only a octeontx2 network driver from octeontx2 common driver
> > > i.e application should not expect any stability on intra driver functions or it does not meant to
> > > be used by application.
> > > 
> > Ok, but again, your assertion is that its driver to driver only, but in
> > practicaility, that assertion is irrelevant.  Those symbols are still exposed
> > for general use, and weather or not you say they aren't part of the ABI, the
> > fact of the matter is, there is no way to tell the difference from a linked
> > object standpoint.  Instead of hobbling the tool to just not scan anything, you
> > need to find a way to differentiate these symbols, so that you can enforce your
> > assertion that there are restrictions on where these APIs are called from.
> > 
> > Neil
> > 
> > > Thomas,
> > > Any thought on this?
> 
> As Neil said, we need to differentiate the internal APIs.
> We already have this issue in a number of places like EAL, or ethdev,
> and it was poorly addressed with some comments like "@internal".
> 
> Practically I don't care about stability of these internal functions,
> but I agree that it creates a mess in the tooling and confuse users.
> 
Agreed, If we can mark them in a way that can enforce no outside usage, then
there need not be any ABI compatibility guarantee

Neil

> 
> 

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, May 22, 2019 4:21 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Cc: Neil Horman <nhorman@tuxdriver.com>; dev@dpdk.org;
> thomas@monjalon.net; stable@dpdk.org
> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH] devtools: skip the symbol check
> when map file under drivers
> 
> > > Sorry, but I'm not ok with this, because many of our DPDK PMDs have
> > > functions that get exported which are meant to be called by
> > > applications directly.  The
> >
> > OK. Just to update my knowledge, Should those API needs to go through
> > ABI/API depreciation process?
> >
> > Actually, I am concerned about the APIs, which is called between
> > drviers not the application. For example,
> > drivers/common/dpaax/rte_common_dpaax_version.map
> >
> > it is not interface to application rather it is for intra driver case.
> > I think, I can change my logic to Skip the symbols which NOT starting with
> rte_.
> > Agree?
> >
> > Context:
> > I am adding a new driver/common/octeontx2 directory and it has some
> > API which Needs to shared between drivers not to the application. For
> > me, it does not make sense to go through any ABI process in such case.
> >
> >
> Maybe not, but other drivers will have APIs designed for apps to call directly -
> some NIC drivers have them, and I suspect that rawdev drivers will need
> them a lot. Therefore, it's best to have the drivers directory scanned by our
> tooling.

Agreed. But all of those API  which called directly called from application
is starts with rte_ symbol. How about skipping the symbols which is NOT start with rte_*
example:
drivers/common/octeontx/rte_common_octeontx_version.map
drivers/common/dpaax/rte_common_dpaax_version.map

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Wed, May 22, 2019 at 11:54:13AM +0000, Jerin Jacob Kollanukkaran wrote:
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Wednesday, May 22, 2019 4:21 PM
> > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > Cc: Neil Horman <nhorman@tuxdriver.com>; dev@dpdk.org;
> > thomas@monjalon.net; stable@dpdk.org
> > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH] devtools: skip the symbol check
> > when map file under drivers
> > 
> > > > Sorry, but I'm not ok with this, because many of our DPDK PMDs have
> > > > functions that get exported which are meant to be called by
> > > > applications directly.  The
> > >
> > > OK. Just to update my knowledge, Should those API needs to go through
> > > ABI/API depreciation process?
> > >
> > > Actually, I am concerned about the APIs, which is called between
> > > drviers not the application. For example,
> > > drivers/common/dpaax/rte_common_dpaax_version.map
> > >
> > > it is not interface to application rather it is for intra driver case.
> > > I think, I can change my logic to Skip the symbols which NOT starting with
> > rte_.
> > > Agree?
> > >
> > > Context:
> > > I am adding a new driver/common/octeontx2 directory and it has some
> > > API which Needs to shared between drivers not to the application. For
> > > me, it does not make sense to go through any ABI process in such case.
> > >
> > >
> > Maybe not, but other drivers will have APIs designed for apps to call directly -
> > some NIC drivers have them, and I suspect that rawdev drivers will need
> > them a lot. Therefore, it's best to have the drivers directory scanned by our
> > tooling.
> 
> Agreed. But all of those API  which called directly called from application
> is starts with rte_ symbol. How about skipping the symbols which is NOT start with rte_*
> example:
> drivers/common/octeontx/rte_common_octeontx_version.map
> drivers/common/dpaax/rte_common_dpaax_version.map
> 

No, that won't work.  If you export a function, it doesn't matter if its named
rte_* or not.  Its accessible from any library/application that cares to call
it, and so you have a responsibility to keep it stable for those users.

Currently the way we have around that is the use of the __rte_experimental tag.
Adding that tag to an exported function marks it as being unstable, and while
you can use it, it will generate a build time warning about its use, unless you
define ALLOW_EXPERIMENTAL_API.  You could use that, understanding that in-tree
drivers could use it safely, as you should always be keeping the API in sync
with its users, but thats not quite what you want I don't think.

Another solution (allbeit a slightly risky one), would be to bifurcate your
header files into a public and private version, with the private version
prototyping your driver-only functions properly, and the public version aliasing
them such that they generate a build time error indicating those functions
aren't available for public use (you can use the gcc static_assert macro I
believe).  Users could circumvent it by pulling the private header out of the
build, or just prototyping the functions themselves, but at that point a user is
asking for trouble anyway

Neil

[dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[jerinj]

From: Jerin Jacob <jerinj@marvell.com>

Drivers do not have ABI.
Skip the symbol check if map file under drivers directory.

Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")

Cc: stable@dpdk.org
Cc: Neil Horman <nhorman@tuxdriver.com>

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
---
 devtools/check-symbol-change.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
index c5434f3bb..444beddad 100755
--- a/devtools/check-symbol-change.sh
+++ b/devtools/check-symbol-change.sh
@@ -93,6 +93,14 @@ check_for_rule_violations()
 		if [ "$ar" = "add" ]
 		then
 
+			directory=`echo $mname | cut -d / -f 2`
+			if [ "$directory" = "drivers" ]
+			then
+				# Drivers do not have ABI. Skip further
+				# processing if the map file is under
+				# drivers directory
+				continue
+			fi
 			if [ "$secname" = "unknown" ]
 			then
 				# Just inform the user of this occurrence, but
-- 
2.21.0

Re: [dpdk-dev] [PATCH] devtools: skip the symbol check when map file under drivers

[Neil Horman]

On Wed, May 22, 2019 at 01:26:28AM +0530, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
> 
> Drivers do not have ABI.
> Skip the symbol check if map file under drivers directory.
> 
> Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> 
> Cc: stable@dpdk.org
> Cc: Neil Horman <nhorman@tuxdriver.com>
> 
Sorry, but I'm not ok with this, because many of our DPDK PMDs have functions
that get exported which are meant to be called by applications directly.  The
dpaa2 driver is a good example, the cryptodev scheduler is another.  Take a look
at their version.map files to see what I mean.

Unless we are willing to make drivers opaque objects that are only accessible
from the [eth|crypto|etc]dev apis in the DPDK core, we have the potential for
exported symbols, which means we have an ABI that has to be maintained, or at
least recognized and reviewed for consistency

Nacked-by: Neil Horman <nhorman@tuxdriver.com>


> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> ---
>  devtools/check-symbol-change.sh | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
> index c5434f3bb..444beddad 100755
> --- a/devtools/check-symbol-change.sh
> +++ b/devtools/check-symbol-change.sh
> @@ -93,6 +93,14 @@ check_for_rule_violations()
>  		if [ "$ar" = "add" ]
>  		then
>  
> +			directory=`echo $mname | cut -d / -f 2`
> +			if [ "$directory" = "drivers" ]
> +			then
> +				# Drivers do not have ABI. Skip further
> +				# processing if the map file is under
> +				# drivers directory
> +				continue
> +			fi
>  			if [ "$secname" = "unknown" ]
>  			then
>  				# Just inform the user of this occurrence, but
> -- 
> 2.21.0
> 
>