* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 10:54 ` [dpdk-techboard] " Bruce Richardson
@ 2019-04-04 12:02 ` Luca Boccassi
2019-04-04 13:05 ` Ray Kinsella
2019-04-04 12:52 ` Ray Kinsella
` (2 subsequent siblings)
3 siblings, 1 reply; 39+ messages in thread
From: Luca Boccassi @ 2019-04-04 12:02 UTC (permalink / raw)
To: Bruce Richardson, Burakov, Anatoly
Cc: Ray Kinsella, dev, Kevin Traynor, techboard
On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> > On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> > > Hi folks,
> > >
> > > Recently I started a discussion with the DPDK Technical Board on
> > > DPDK
> > > ABI/API stability. This was born out informal feedback I had
> > > received
> > > from a number of users of DPDK about ABI churn. In turn this
> > > feedback
> > > then prompted an ABI analysis of DPDK using tools from abi-
> > > laboratory.
> > >
> > > https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> > >
> > >
> > > I guess the short story is that DPDK ABI hasn't really settled
> > > down as
> > > the project has matured. If you take a look at the “Backward
> > > Compat.”
> > > column which measures ABI compatibility compared to the previous
> > > releases, you will see significant churn in the ABI over
> > > successive
> > > releases since v16.04.
> > >
> > > Now compare DPDK to GStreamer as an example of a very mature
> > > project
> > > with a similar intent, a framework for building applications, and
> > > which
> > > enjoys a very stable API.
> > >
> > > https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> > >
> > >
> > > The DPDK ABI churn has the following affects for users:-
> > >
> > > 1. The churn obliges users of DPDK to commit to a constant
> > > re-integration and re-validation effort for new versions of DPDK.
> > > This
> > > effort from their perspective may not add value to their
> > > consuming
> > > project, particular if they are only updating to "stay current".
> > > 2. The churn encourages users of DPDK to slip versions, putting
> > > off
> > > reintegration to later, building up technical debt and causing
> > > their
> > > projects to miss support for new hardware or features.
> > > 3. It makes DPDK different to almost every other system library
> > > and
> > > framework that an operating systems might ship. This makes DPDK
> > > trickier
> > > to dynamically link against, package and maintain for OS
> > > maintainers.
> > >
> > > In order to address this issue, I have put together the minimal
> > > set of
> > > concrete proposals below for discussion at the Technical Board
> > > next
> > > Wednesday.
> > >
> > > I wanted to share this, as these might not yet be the right
> > > proposals,
> > > however I am putting them out there for feedback to start the
> > > discussion.
> > >
> > > Thanks,
> > >
> > > Ray K
> > >
> > >
> > > Experimental API
> > > 1. APIs designated as experimental are not considered part
> > > of the ABI
> > > and may change without warning at any time.
> > > 2. APIs designated as experimental must be marked
> > > depreciated for a
> > > least one quarterly release before removal.
> > > 3. APIs designated as experimental will no longer
> > > automatically graduate
> > > to core after one release, they may stay experimental until their
> > > author
> > > and the maintainer agree that graduation is appropriate.
> > >
> > > Core API (non-experimental API)
> > > 4. APIs designated as core must be depreciated for a least
> > > two years
> > > before removal, to facilitate the continued compatibility with
> > > LTS
> > > releases. A final removal notice will be published to the DPDK
> > > Mailing
> > > List, and if there are no strong objections only then an API may
> > > be
> > > removed.
> > > 5. APIs designated as core may be changed as follows:-
> > > 5.a The change proposer must demonstrated that the change
> > > has a
> > > supporting use case and could not be achieved in any other way.
> > > 5.b ABI version compatibility must be retained, as
> > > described below.
> >
> > Hi Ray,
> >
> > My somewhat rambly 2 cents :)
> >
> > While i think some solution has to be found for the situation, we
> > also have
> > to balance this against speed of development and new features
> > rollout.
> >
> > For example, let's consider what i am intimately familiar with -
> > the memory
> > rework. I have made enormous efforts to ensure that pre-18.05 and
> > post-18.05
> > remain as ABI/API compatible as possible, but there were a couple
> > of API
> > calls that were removed, and there couldn't have been any
> > replacements
> > (these API's were exposing internal structures that shouldn't have
> > been
> > exposed in the first place), and 18.05 also broke the ABI
> > compatibility,
> > because there was no way to do it without it (shared internal
> > structures
> > needed to change in part to support multiprocess).
> >
> > So, if i understand your proposal correctly, assuming a 2-year
> > waiting
> > period for the deprecation of core API's, you would essentially
> > still be
> > waiting for the memory rework to land for a year more. Moreover,
> > even
> > *after* it has landed, there was a continuous stream of
> > improvements and
> > bugfixes, some of which has broke ABI compatibility as well. Some
> > of them
> > were my fault (as in, i could've foreseen the need for those
> > changes, but
> > didn't), but others came as a result of people using these new
> > features in
> > the wild and reporting issues/problems/suggestions - i am but one
> > man, after
> > all. Plus, you know, there's only 24 hours in a day, and some stuff
> > takes
> > time to implement :)
> >
> > Since this rework goes right at the heart of DPDK (arguably there
> > isn't a
> > more "core" API than memory!), there is no (sane) way in the
> > universe to 1)
> > keep backwards compatibility for this, or 2) keep two parallel
> > versions of
> > it. We also need to test all that, and, to be honest, one
> > validation cycle
> > for a release wouldn't be enough to figure out all of the kinks and
> > implications of such a case. It was really great that memory rework
> > has
> > landed in 18.05 and we had time to improve and prepare it for an
> > 18.11 LTS -
> > i think everyone can say that it's in much better shape in 18.11
> > than it was
> > in 18.05, but if we couldn't do an ABI break here or there, this
> > rate of
> > improvements would have slowed down significantly.
> >
> > Now, i understand that this is probably a highly exceptional case,
> > but i'm
> > sure that maintainers of other parts of DPDK will have their own
> > examples of
> > similar things happening.
> >
> > I have no idea what a proper solution would look like. Any
> > "splitting" of
> > the trees into "experimental" vs. "stable" will end up causing the
> > same
> > issue - people choose to use stable over experimental because,
> > well, it's
> > more stable, and new/experimental features don't get tested as much
> > because
> > no one runs the thing in the first place.
> >
> > TL;DR we have to be careful not to constrain the pace of
> > development/bugfixing just for the sake of having a stable API/ABI
> > :)
> >
>
> Actually, I think we *do* need to constrain the pace of development
> for the
> sake of ABI stability. At this stage DPDK has been around for quite a
> number of years and so should be considered a fairly mature project -
> it
> should just start acting like it.
>
> Now, in terms of features like the memory rework, that is indeed a
> case
> that there was no alternative other than a massive ABI break.
> However, for
> that rework there was a strong need for improvement in that area that
> we
> can make the case for an ABI break to support it - and it is of a
> scale
> that nothing other than an ABI change would do. For other areas and
> examples, I doubt there are many in the last couple of years that are
> of
> that scale.
Fully agree.
It's normal for new project, big and small, to start without a
stability promise as development ramps up, and then steer toward
stability as the user base grows. Sometimes the switch is made explicit
by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
I don't think we crossed that boundary yet in this project, and I
believe that's the main question Ray was trying to raise: has the time
finally come for DPDK to do this phase shift?
Of course it comes with a price for all developers, and that's again
common.
> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should
> reduce
> the scope for ABI breaks.
Yes, I'm a big fan of accessors and opaque structs.
> 2. Once done, I think we should commit to having an ABI break only in
> the
> rarest of circumstances, and only with very large justification. I
> want us
> to get to the point where DPDK releases can immediately be picked up
> by all
> linux distros and rolled out because they are ABI compatible.
>
> I'm not sure I like the idea of planned ABI break releases - that
> strikes
> me as a plan where we end up with the same number of ABI breaks as
> before,
> just balled into one release.
I think that was intended as a compromise, especially as we move from
one model to the other, and more of a "if a breakage has to happen, it
must be in the X release" rather than "let's always break in the X
release" :-)
> Question for Kevin, Luca and others who look at distro-packaging: is
> it the
> case that each distro will only ship one version of DPDK, or is it
> possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
We can ship multiple versions, although it's more work so there should
be a good reason to do it. At the moment in Debian and Ubuntu we don't,
and we tend to ship whatever the latest LTS version is at the distro
freeze milestone - for example Debian 10 which will be released soon
(TM) will have 18.11.0.
> So, in short, I'm generally in favour of a zero-tolerance approach
> for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory
> layout or
> similar.
>
> Regards,
> /Bruce
>
--
Kind regards,
Luca Boccassi
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 12:02 ` Luca Boccassi
@ 2019-04-04 13:05 ` Ray Kinsella
2019-04-04 13:10 ` Bruce Richardson
2019-04-04 13:21 ` Luca Boccassi
0 siblings, 2 replies; 39+ messages in thread
From: Ray Kinsella @ 2019-04-04 13:05 UTC (permalink / raw)
To: Luca Boccassi, Bruce Richardson, Burakov, Anatoly
Cc: dev, Kevin Traynor, techboard
On 04/04/2019 13:02, Luca Boccassi wrote:
> On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
[SNIP]
>>>
>>
>> Actually, I think we *do* need to constrain the pace of development
>> for the
>> sake of ABI stability. At this stage DPDK has been around for quite a
>> number of years and so should be considered a fairly mature project -
>> it
>> should just start acting like it.
>>
>> Now, in terms of features like the memory rework, that is indeed a
>> case
>> that there was no alternative other than a massive ABI break.
>> However, for
>> that rework there was a strong need for improvement in that area that
>> we
>> can make the case for an ABI break to support it - and it is of a
>> scale
>> that nothing other than an ABI change would do. For other areas and
>> examples, I doubt there are many in the last couple of years that are
>> of
>> that scale.
>
> Fully agree.
>
> It's normal for new project, big and small, to start without a
> stability promise as development ramps up, and then steer toward
> stability as the user base grows. Sometimes the switch is made explicit
> by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
> I don't think we crossed that boundary yet in this project, and I
> believe that's the main question Ray was trying to raise: has the time
> finally come for DPDK to do this phase shift?
Yes - we never had a 1.0, where we cut an API and stood behind it
similar to GStreamer. There a number of reasons why this happened not
worth going into, however you make the point very well Luca - this phase
shift is long over due.
>
> Of course it comes with a price for all developers, and that's again
> common.
Agreed - nothing is for free.
The question is this something we value and is it something worth doing.
>
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should
>> reduce
>> the scope for ABI breaks.
>
> Yes, I'm a big fan of accessors and opaque structs.
+1, me too.
>
>> 2. Once done, I think we should commit to having an ABI break only in
>> the
>> rarest of circumstances, and only with very large justification. I
>> want us
>> to get to the point where DPDK releases can immediately be picked up
>> by all
>> linux distros and rolled out because they are ABI compatible.
>>
>> I'm not sure I like the idea of planned ABI break releases - that
>> strikes
>> me as a plan where we end up with the same number of ABI breaks as
>> before,
>> just balled into one release.
>
> I think that was intended as a compromise, especially as we move from
> one model to the other, and more of a "if a breakage has to happen, it
> must be in the X release" rather than "let's always break in the X
> release" :-)
>
>> Question for Kevin, Luca and others who look at distro-packaging: is
>> it the
>> case that each distro will only ship one version of DPDK, or is it
>> possible
>> that if we have ABI breaks, a distro will provide two copies of DPDK
>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>
> We can ship multiple versions, although it's more work so there should
> be a good reason to do it.
Well you already kind of do right.
You ship 16.11.8 with Debian 9 and then 18.11 with Debian 9 backports.
> At the moment in Debian and Ubuntu we don't,
> and we tend to ship whatever the latest LTS version is at the distro
> freeze milestone - for example Debian 10 which will be released soon
> (TM) will have 18.11.0.
Presumably when 19.11 arrives, it will land in Debian 10 backports
similarly.
I assume anything that lands in backports is not guaranteed to be ABI
compatible with stable?
>
>> So, in short, I'm generally in favour of a zero-tolerance approach
>> for DPDK
>> ABI breaks, and having ABI breaks as a major event reserved only for
>> massive rework changes, such as major mbuf changes, or new memory
>> layout or
>> similar.
>>
>> Regards,
>> /Bruce
>>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 13:05 ` Ray Kinsella
@ 2019-04-04 13:10 ` Bruce Richardson
2019-04-05 13:25 ` [dpdk-dev] " Ray Kinsella
2019-04-04 13:21 ` Luca Boccassi
1 sibling, 1 reply; 39+ messages in thread
From: Bruce Richardson @ 2019-04-04 13:10 UTC (permalink / raw)
To: Ray Kinsella
Cc: Luca Boccassi, Burakov, Anatoly, dev, Kevin Traynor, techboard
On Thu, Apr 04, 2019 at 02:05:27PM +0100, Ray Kinsella wrote:
>
>
> On 04/04/2019 13:02, Luca Boccassi wrote:
> > On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> >> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> [SNIP]
> >>>
> >>
> >> Actually, I think we *do* need to constrain the pace of development
> >> for the
> >> sake of ABI stability. At this stage DPDK has been around for quite a
> >> number of years and so should be considered a fairly mature project -
> >> it
> >> should just start acting like it.
> >>
> >> Now, in terms of features like the memory rework, that is indeed a
> >> case
> >> that there was no alternative other than a massive ABI break.
> >> However, for
> >> that rework there was a strong need for improvement in that area that
> >> we
> >> can make the case for an ABI break to support it - and it is of a
> >> scale
> >> that nothing other than an ABI change would do. For other areas and
> >> examples, I doubt there are many in the last couple of years that are
> >> of
> >> that scale.
> >
> > Fully agree.
> >
> > It's normal for new project, big and small, to start without a
> > stability promise as development ramps up, and then steer toward
> > stability as the user base grows. Sometimes the switch is made explicit
> > by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
> > I don't think we crossed that boundary yet in this project, and I
> > believe that's the main question Ray was trying to raise: has the time
> > finally come for DPDK to do this phase shift?
>
> Yes - we never had a 1.0, where we cut an API and stood behind it
> similar to GStreamer. There a number of reasons why this happened not
> worth going into, however you make the point very well Luca - this phase
> shift is long over due.
>
> >
> > Of course it comes with a price for all developers, and that's again
> > common.
>
> Agreed - nothing is for free.
> The question is this something we value and is it something worth doing.
>
> >
> >> My thoughts on the matter are:
> >> 1. I think we really need to do work to start hiding more of our data
> >> structures - like what Stephen's latest RFC does. This hiding should
> >> reduce
> >> the scope for ABI breaks.
> >
> > Yes, I'm a big fan of accessors and opaque structs.
>
> +1, me too.
>
I would be too, with certain exceptions - rte_mbuf for one. Any structures
that are used by applications cannot be made opaque, as apps do not want to
pay the cost of having to call a function every time they want to access
something from one of those structures.
Thankfully for us, we have plenty of other structures that we can work on
in the meantime that can be made private to avoid ABI breaks! :-) I suggest
we work through those first, allowing us to hone our ABI-break avoidance
skills.
/Bruce
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 13:10 ` Bruce Richardson
@ 2019-04-05 13:25 ` Ray Kinsella
2019-04-07 9:37 ` Thomas Monjalon
0 siblings, 1 reply; 39+ messages in thread
From: Ray Kinsella @ 2019-04-05 13:25 UTC (permalink / raw)
To: Bruce Richardson
Cc: Luca Boccassi, Burakov, Anatoly, dev, Kevin Traynor, techboard
On 04/04/2019 14:10, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 02:05:27PM +0100, Ray Kinsella wrote:
>>
>>
>> On 04/04/2019 13:02, Luca Boccassi wrote:
>>> On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>
> I would be too, with certain exceptions - rte_mbuf for one. Any structures
> that are used by applications cannot be made opaque, as apps do not want to
> pay the cost of having to call a function every time they want to access
> something from one of those structures.
These the layout of these structures really must become sacrosanct.
As Stephen points out, there may be room for a one more change - fool me
once - to future poof the structure but after, that these structure will
become very hard to change.
>
> Thankfully for us, we have plenty of other structures that we can work on
> in the meantime that can be made private to avoid ABI breaks! :-) I suggest
> we work through those first, allowing us to hone our ABI-break avoidance
> skills.
>
> /Bruce
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-05 13:25 ` [dpdk-dev] " Ray Kinsella
@ 2019-04-07 9:37 ` Thomas Monjalon
0 siblings, 0 replies; 39+ messages in thread
From: Thomas Monjalon @ 2019-04-07 9:37 UTC (permalink / raw)
To: techboard
Cc: Ray Kinsella, Bruce Richardson, Luca Boccassi, Burakov, Anatoly,
dev, Kevin Traynor
05/04/2019 15:25, Ray Kinsella:
> On 04/04/2019 14:10, Bruce Richardson wrote:
> > On Thu, Apr 04, 2019 at 02:05:27PM +0100, Ray Kinsella wrote:
> >> On 04/04/2019 13:02, Luca Boccassi wrote:
> >>> On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> >>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> >>
> > I would be too, with certain exceptions - rte_mbuf for one. Any structures
> > that are used by applications cannot be made opaque, as apps do not want to
> > pay the cost of having to call a function every time they want to access
> > something from one of those structures.
>
> These the layout of these structures really must become sacrosanct.
> As Stephen points out, there may be room for a one more change - fool me
> once - to future poof the structure but after, that these structure will
> become very hard to change.
Yes, looks like a good plan.
> > Thankfully for us, we have plenty of other structures that we can work on
> > in the meantime that can be made private to avoid ABI breaks! :-) I suggest
> > we work through those first, allowing us to hone our ABI-break avoidance
> > skills.
+1 to work on obvious cases first.
I think we should not wait too much to improve mbuf
with some dynamically registered fields, because it may be
long to achieve such an important feature and make everybody happy.
Note: we may have to sacrifice a few cycles for some features
like distributor or IPsec.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 13:05 ` Ray Kinsella
2019-04-04 13:10 ` Bruce Richardson
@ 2019-04-04 13:21 ` Luca Boccassi
1 sibling, 0 replies; 39+ messages in thread
From: Luca Boccassi @ 2019-04-04 13:21 UTC (permalink / raw)
To: Ray Kinsella, Bruce Richardson, Burakov, Anatoly
Cc: dev, Kevin Traynor, techboard
On Thu, 2019-04-04 at 14:05 +0100, Ray Kinsella wrote:
> > > Question for Kevin, Luca and others who look at distro-packaging:
> > > is
> > > it the
> > > case that each distro will only ship one version of DPDK, or is
> > > it
> > > possible
> > > that if we have ABI breaks, a distro will provide two copies of
> > > DPDK
> > > simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
> >
> > We can ship multiple versions, although it's more work so there
> > should
> > be a good reason to do it.
>
> Well you already kind of do right.
> You ship 16.11.8 with Debian 9 and then 18.11 with Debian 9
> backports.
Yes indeed - backports has a different policy and is not part of
"Debian main" - for example, stability and security support are not
guaranteed. Also there is no guarantee of compatibility of co-
installability - while you can have the individual libraries from each
version at the same time, you can't have both libdpdk-dev packages
which are necessary to build an application. So you have to choose and
install either libdpdk-dev=16.11 or libdpdk-dev=18.11.
It is possible to have multiple versions in the same archive at the
same time, and with multiple -dev packages, and that's what I meant
when I mentioned it was more work.
> > At the moment in Debian and Ubuntu we don't,
> > and we tend to ship whatever the latest LTS version is at the
> > distro
> > freeze milestone - for example Debian 10 which will be released
> > soon
> > (TM) will have 18.11.0.
>
> Presumably when 19.11 arrives, it will land in Debian 10 backports
> similarly.
>
> I assume anything that lands in backports is not guaranteed to be ABI
> compatible with stable?
Yes, there's no guarantee.
--
Kind regards,
Luca Boccassi
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 10:54 ` [dpdk-techboard] " Bruce Richardson
2019-04-04 12:02 ` Luca Boccassi
@ 2019-04-04 12:52 ` Ray Kinsella
2019-04-04 14:07 ` Burakov, Anatoly
2019-04-04 15:51 ` Stephen Hemminger
2019-04-04 16:56 ` Kevin Traynor
3 siblings, 1 reply; 39+ messages in thread
From: Ray Kinsella @ 2019-04-04 12:52 UTC (permalink / raw)
To: Bruce Richardson, Burakov, Anatoly; +Cc: dev, Kevin Traynor, techboard
On 04/04/2019 11:54, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>> Hi folks,
>>>
[SNIP]
>>
>> Hi Ray,
>>
>> My somewhat rambly 2 cents :)
>>
>> While i think some solution has to be found for the situation, we also have
>> to balance this against speed of development and new features rollout.
>>
>> For example, let's consider what i am intimately familiar with - the memory
>> rework. I have made enormous efforts to ensure that pre-18.05 and post-18.05
>> remain as ABI/API compatible as possible, but there were a couple of API
>> calls that were removed, and there couldn't have been any replacements
>> (these API's were exposing internal structures that shouldn't have been
>> exposed in the first place), and 18.05 also broke the ABI compatibility,
>> because there was no way to do it without it (shared internal structures
>> needed to change in part to support multiprocess).
>>
>> So, if i understand your proposal correctly, assuming a 2-year waiting
>> period for the deprecation of core API's, you would essentially still be
>> waiting for the memory rework to land for a year more. Moreover, even
>> *after* it has landed, there was a continuous stream of improvements and
>> bugfixes, some of which has broke ABI compatibility as well. Some of them
>> were my fault (as in, i could've foreseen the need for those changes, but
>> didn't), but others came as a result of people using these new features in
>> the wild and reporting issues/problems/suggestions - i am but one man, after
>> all. Plus, you know, there's only 24 hours in a day, and some stuff takes
>> time to implement :)
>>
>> Since this rework goes right at the heart of DPDK (arguably there isn't a
>> more "core" API than memory!), there is no (sane) way in the universe to 1)
>> keep backwards compatibility for this, or 2) keep two parallel versions of
>> it. We also need to test all that, and, to be honest, one validation cycle
>> for a release wouldn't be enough to figure out all of the kinks and
>> implications of such a case. It was really great that memory rework has
>> landed in 18.05 and we had time to improve and prepare it for an 18.11 LTS -
>> i think everyone can say that it's in much better shape in 18.11 than it was
>> in 18.05, but if we couldn't do an ABI break here or there, this rate of
>> improvements would have slowed down significantly.
>>
>> Now, i understand that this is probably a highly exceptional case, but i'm
>> sure that maintainers of other parts of DPDK will have their own examples of
>> similar things happening.
>>
>> I have no idea what a proper solution would look like. Any "splitting" of
>> the trees into "experimental" vs. "stable" will end up causing the same
>> issue - people choose to use stable over experimental because, well, it's
>> more stable, and new/experimental features don't get tested as much because
>> no one runs the thing in the first place.
>>
>> TL;DR we have to be careful not to constrain the pace of
>> development/bugfixing just for the sake of having a stable API/ABI :)
>>
>
> Actually, I think we *do* need to constrain the pace of development for the
> sake of ABI stability. At this stage DPDK has been around for quite a
> number of years and so should be considered a fairly mature project - it
> should just start acting like it.
I 100% agree.
If you break your users stuff regularly enough, they will eventually
start looking around for an alternative that doesn't break their stuff
quiet so regularly.
We often use the pace of innovation in DPDK as justification for ABI/API
breakages, but that approach is a real rarity among the Open Source
community. I can't think of any mature project off-hand that share's it.
I would ask is Linux any less innovative because they offer a stable API
and have an absolute commitment to never breaking userspace? Would Linux
have ever been as popular as it is today it they broke userspace every
quarter?
They reality is that they (Linux) find workarounds and compromise
because there is an uber-maintainer Linus who had a strong ethos from
the start not to break their users stuff - we need the same ethos in DPDK.
>
> Now, in terms of features like the memory rework, that is indeed a case
> that there was no alternative other than a massive ABI break. However, for
> that rework there was a strong need for improvement in that area that we
> can make the case for an ABI break to support it - and it is of a scale
> that nothing other than an ABI change would do. For other areas and
> examples, I doubt there are many in the last couple of years that are of
> that scale.
I would also be inclined to agree with Bruce's points on memory rework
was somewhat of an outlier, we don't see many like it.
> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should reduce
> the scope for ABI breaks.
> 2. Once done, I think we should commit to having an ABI break only in the
> rarest of circumstances, and only with very large justification. I want us
> to get to the point where DPDK releases can immediately be picked up by all
> linux distros and rolled out because they are ABI compatible.
The work that Anatoly describes removing APIs that exposed internal
structures and Stephen H's RFC similarly are good examples of the kind
of work required to prepare for this change. We need to take a good look
at the API and reduce the number of unnecessary internal structures
exposed.
I never expected it going to to be a big bang - but is a definite
direction we need to move towards over the next few release.
> I'm not sure I like the idea of planned ABI break releases - that strikes
> me as a plan where we end up with the same number of ABI breaks as before,
> just balled into one release.
>
I agree, with this approach you just delaying the evil day and when that
day comes it will 10-times more painful. This is no substitute for a
don't break our users stuff ethos.
> Question for Kevin, Luca and others who look at distro-packaging: is it the
> case that each distro will only ship one version of DPDK, or is it possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>
>
> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory layout or
> similar.
>
> Regards,
> /Bruce
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 12:52 ` Ray Kinsella
@ 2019-04-04 14:07 ` Burakov, Anatoly
2019-04-07 9:48 ` [dpdk-dev] " Thomas Monjalon
0 siblings, 1 reply; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-04 14:07 UTC (permalink / raw)
To: Ray Kinsella, Bruce Richardson; +Cc: dev, Kevin Traynor, techboard
On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>
>
> On 04/04/2019 11:54, Bruce Richardson wrote:
>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>>> Hi folks,
>>>>
> [SNIP]
>>>
>>> Hi Ray,
>>>
>>> My somewhat rambly 2 cents :)
>>>
>>> While i think some solution has to be found for the situation, we also have
>>> to balance this against speed of development and new features rollout.
>>>
>>> For example, let's consider what i am intimately familiar with - the memory
>>> rework. I have made enormous efforts to ensure that pre-18.05 and post-18.05
>>> remain as ABI/API compatible as possible, but there were a couple of API
>>> calls that were removed, and there couldn't have been any replacements
>>> (these API's were exposing internal structures that shouldn't have been
>>> exposed in the first place), and 18.05 also broke the ABI compatibility,
>>> because there was no way to do it without it (shared internal structures
>>> needed to change in part to support multiprocess).
>>>
>>> So, if i understand your proposal correctly, assuming a 2-year waiting
>>> period for the deprecation of core API's, you would essentially still be
>>> waiting for the memory rework to land for a year more. Moreover, even
>>> *after* it has landed, there was a continuous stream of improvements and
>>> bugfixes, some of which has broke ABI compatibility as well. Some of them
>>> were my fault (as in, i could've foreseen the need for those changes, but
>>> didn't), but others came as a result of people using these new features in
>>> the wild and reporting issues/problems/suggestions - i am but one man, after
>>> all. Plus, you know, there's only 24 hours in a day, and some stuff takes
>>> time to implement :)
>>>
>>> Since this rework goes right at the heart of DPDK (arguably there isn't a
>>> more "core" API than memory!), there is no (sane) way in the universe to 1)
>>> keep backwards compatibility for this, or 2) keep two parallel versions of
>>> it. We also need to test all that, and, to be honest, one validation cycle
>>> for a release wouldn't be enough to figure out all of the kinks and
>>> implications of such a case. It was really great that memory rework has
>>> landed in 18.05 and we had time to improve and prepare it for an 18.11 LTS -
>>> i think everyone can say that it's in much better shape in 18.11 than it was
>>> in 18.05, but if we couldn't do an ABI break here or there, this rate of
>>> improvements would have slowed down significantly.
>>>
>>> Now, i understand that this is probably a highly exceptional case, but i'm
>>> sure that maintainers of other parts of DPDK will have their own examples of
>>> similar things happening.
>>>
>>> I have no idea what a proper solution would look like. Any "splitting" of
>>> the trees into "experimental" vs. "stable" will end up causing the same
>>> issue - people choose to use stable over experimental because, well, it's
>>> more stable, and new/experimental features don't get tested as much because
>>> no one runs the thing in the first place.
>>>
>>> TL;DR we have to be careful not to constrain the pace of
>>> development/bugfixing just for the sake of having a stable API/ABI :)
>>>
>>
>> Actually, I think we *do* need to constrain the pace of development for the
>> sake of ABI stability. At this stage DPDK has been around for quite a
>> number of years and so should be considered a fairly mature project - it
>> should just start acting like it.
>
> I 100% agree.
>
> If you break your users stuff regularly enough, they will eventually
> start looking around for an alternative that doesn't break their stuff
> quiet so regularly.
>
> We often use the pace of innovation in DPDK as justification for ABI/API
> breakages, but that approach is a real rarity among the Open Source
> community. I can't think of any mature project off-hand that share's it.
>
> I would ask is Linux any less innovative because they offer a stable API
> and have an absolute commitment to never breaking userspace? Would Linux
> have ever been as popular as it is today it they broke userspace every
> quarter?
>
> They reality is that they (Linux) find workarounds and compromise
> because there is an uber-maintainer Linus who had a strong ethos from
> the start not to break their users stuff - we need the same ethos in DPDK.
>
>>
>> Now, in terms of features like the memory rework, that is indeed a case
>> that there was no alternative other than a massive ABI break. However, for
>> that rework there was a strong need for improvement in that area that we
>> can make the case for an ABI break to support it - and it is of a scale
>> that nothing other than an ABI change would do. For other areas and
>> examples, I doubt there are many in the last couple of years that are of
>> that scale.
>
> I would also be inclined to agree with Bruce's points on memory rework
> was somewhat of an outlier, we don't see many like it.
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should reduce
>> the scope for ABI breaks.
>> 2. Once done, I think we should commit to having an ABI break only in the
>> rarest of circumstances, and only with very large justification. I want us
>> to get to the point where DPDK releases can immediately be picked up by all
>> linux distros and rolled out because they are ABI compatible.
>
> The work that Anatoly describes removing APIs that exposed internal
> structures and Stephen H's RFC similarly are good examples of the kind
> of work required to prepare for this change. We need to take a good look
> at the API and reduce the number of unnecessary internal structures
> exposed.
>
> I never expected it going to to be a big bang - but is a definite
> direction we need to move towards over the next few release.
...in this case, we have to think long and hard about the fabled EAL
rework/split, and in general *specifying* what is it that we want to
support, and the use cases that we want to target. Right now there is a
huge mountain of technical debt and kludges and workarounds that has
accumulated over the years, and it exists precisely because "every
change breaks someone's workflow".
For example, just in memory subsystem alone, we have legacy mem, because
some use cases require huge amounts of contiguous memory, and not
everyone is using VFIO; there's all of the 32-bit related workarounds
and hacks; there's the single-file-segments stuff that could have been
the default if not for the fact that we support kernels that don't
support fallocate(); there are two different ways of doing in-memory
mode, because not all kernels support memfd's; there is a gargantuan
pile of workarounds (and "known issues", and just code in general) all
over the DPDK codebase just to support our multiprocess model and all of
the various warts that come with it.
In fact, i would even go as far as to say that *most* of EAL ABI breaks
have been due to the fact that we store data in shared memory because of
multiprocess - so there is simply no way we can change these internal
data structures without ABI breaks, because even if they're not exposed
through user-facing API, they are still exposed by virtue of secondary
processes basically having an ABI contract with primary process instances.
So, if we are to cement our core API - we have to make a concrete effort
to specify what goes and what stays, if we want it to be maintainable.
The DPDK 1.0 specification, if you will :)
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 14:07 ` Burakov, Anatoly
@ 2019-04-07 9:48 ` Thomas Monjalon
2019-04-08 9:04 ` Ray Kinsella
0 siblings, 1 reply; 39+ messages in thread
From: Thomas Monjalon @ 2019-04-07 9:48 UTC (permalink / raw)
To: Burakov, Anatoly
Cc: techboard, Ray Kinsella, Bruce Richardson, dev, Kevin Traynor
04/04/2019 16:07, Burakov, Anatoly:
> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> > On 04/04/2019 11:54, Bruce Richardson wrote:
> >> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> >> Actually, I think we *do* need to constrain the pace of development for the
> >> sake of ABI stability. At this stage DPDK has been around for quite a
> >> number of years and so should be considered a fairly mature project - it
> >> should just start acting like it.
> >
> > I 100% agree.
> >
> > If you break your users stuff regularly enough, they will eventually
> > start looking around for an alternative that doesn't break their stuff
> > quiet so regularly.
> >
> > We often use the pace of innovation in DPDK as justification for ABI/API
> > breakages, but that approach is a real rarity among the Open Source
> > community. I can't think of any mature project off-hand that share's it.
> >
> > I would ask is Linux any less innovative because they offer a stable API
> > and have an absolute commitment to never breaking userspace? Would Linux
> > have ever been as popular as it is today it they broke userspace every
> > quarter?
> >
> > They reality is that they (Linux) find workarounds and compromise
> > because there is an uber-maintainer Linus who had a strong ethos from
> > the start not to break their users stuff - we need the same ethos in DPDK.
> >
> >>
> >> Now, in terms of features like the memory rework, that is indeed a case
> >> that there was no alternative other than a massive ABI break. However, for
> >> that rework there was a strong need for improvement in that area that we
> >> can make the case for an ABI break to support it - and it is of a scale
> >> that nothing other than an ABI change would do. For other areas and
> >> examples, I doubt there are many in the last couple of years that are of
> >> that scale.
> >
> > I would also be inclined to agree with Bruce's points on memory rework
> > was somewhat of an outlier, we don't see many like it.
> >> My thoughts on the matter are:
> >> 1. I think we really need to do work to start hiding more of our data
> >> structures - like what Stephen's latest RFC does. This hiding should reduce
> >> the scope for ABI breaks.
> >> 2. Once done, I think we should commit to having an ABI break only in the
> >> rarest of circumstances, and only with very large justification. I want us
> >> to get to the point where DPDK releases can immediately be picked up by all
> >> linux distros and rolled out because they are ABI compatible.
> >
> > The work that Anatoly describes removing APIs that exposed internal
> > structures and Stephen H's RFC similarly are good examples of the kind
> > of work required to prepare for this change. We need to take a good look
> > at the API and reduce the number of unnecessary internal structures
> > exposed.
> >
> > I never expected it going to to be a big bang - but is a definite
> > direction we need to move towards over the next few release.
>
> ...in this case, we have to think long and hard about the fabled EAL
> rework/split, and in general *specifying* what is it that we want to
> support, and the use cases that we want to target. Right now there is a
> huge mountain of technical debt and kludges and workarounds that has
> accumulated over the years, and it exists precisely because "every
> change breaks someone's workflow".
>
> For example, just in memory subsystem alone, we have legacy mem, because
> some use cases require huge amounts of contiguous memory, and not
> everyone is using VFIO; there's all of the 32-bit related workarounds
> and hacks; there's the single-file-segments stuff that could have been
> the default if not for the fact that we support kernels that don't
> support fallocate(); there are two different ways of doing in-memory
> mode, because not all kernels support memfd's; there is a gargantuan
> pile of workarounds (and "known issues", and just code in general) all
> over the DPDK codebase just to support our multiprocess model and all of
> the various warts that come with it.
>
> In fact, i would even go as far as to say that *most* of EAL ABI breaks
> have been due to the fact that we store data in shared memory because of
> multiprocess - so there is simply no way we can change these internal
> data structures without ABI breaks, because even if they're not exposed
> through user-facing API, they are still exposed by virtue of secondary
> processes basically having an ABI contract with primary process instances.
>
> So, if we are to cement our core API - we have to make a concrete effort
> to specify what goes and what stays, if we want it to be maintainable.
> The DPDK 1.0 specification, if you will :)
"DPDK 1.0 specification", that's a great project name :-)
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-07 9:48 ` [dpdk-dev] " Thomas Monjalon
@ 2019-04-08 9:04 ` Ray Kinsella
2019-04-08 10:15 ` Burakov, Anatoly
2019-04-14 0:42 ` Neil Horman
0 siblings, 2 replies; 39+ messages in thread
From: Ray Kinsella @ 2019-04-08 9:04 UTC (permalink / raw)
To: Thomas Monjalon, Burakov, Anatoly
Cc: techboard, Bruce Richardson, dev, Kevin Traynor
On 07/04/2019 10:48, Thomas Monjalon wrote:
> 04/04/2019 16:07, Burakov, Anatoly:
>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
[SNIP]
>> So, if we are to cement our core API - we have to make a concrete effort
>> to specify what goes and what stays, if we want it to be maintainable.
>> The DPDK 1.0 specification, if you will :)
>
> "DPDK 1.0 specification", that's a great project name :-)
>
Honestly - I would say that I am nervous of this.
The definition of a DPDK 1.0 specification as a gate to API stability,
feels like a "great plan tomorrow" instead of a "good plan" today. I
think that getting people to dedicate time to such a specification might
prove problematic and I could see this effort being very time consuming.
It might never get completed.
My preference would be to instead adopt a well-publicised community
timeline for adopting more conservative API maintenance rules.
Perhaps we could give ourselves as a community a time-limited window in
which to address concerns around the API before they become hardened -
perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
to 9 months.
We then would know the timeline when niggles like exposure of internal
structures and mbuf structure needed to be sorted by and could
prioritize accordingly.
Ray K
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 9:04 ` Ray Kinsella
@ 2019-04-08 10:15 ` Burakov, Anatoly
2019-04-08 13:00 ` Ray Kinsella
2019-04-14 0:42 ` Neil Horman
1 sibling, 1 reply; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-08 10:15 UTC (permalink / raw)
To: Ray Kinsella, Thomas Monjalon
Cc: techboard, Bruce Richardson, dev, Kevin Traynor
On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
> On 07/04/2019 10:48, Thomas Monjalon wrote:
>> 04/04/2019 16:07, Burakov, Anatoly:
>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> [SNIP]
>>> So, if we are to cement our core API - we have to make a concrete effort
>>> to specify what goes and what stays, if we want it to be maintainable.
>>> The DPDK 1.0 specification, if you will :)
>>
>> "DPDK 1.0 specification", that's a great project name :-)
>>
>
> Honestly - I would say that I am nervous of this.
>
> The definition of a DPDK 1.0 specification as a gate to API stability,
> feels like a "great plan tomorrow" instead of a "good plan" today. I
> think that getting people to dedicate time to such a specification might
> prove problematic and I could see this effort being very time consuming.
> It might never get completed.
>
> My preference would be to instead adopt a well-publicised community
> timeline for adopting more conservative API maintenance rules.
>
> Perhaps we could give ourselves as a community a time-limited window in
> which to address concerns around the API before they become hardened -
> perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
> to 9 months.
>
> We then would know the timeline when niggles like exposure of internal
> structures and mbuf structure needed to be sorted by and could
> prioritize accordingly.
>
> Ray K
>
My worry here is that some API's get more attention than others, but
requirements for freezing the API/ABI are applicable to all of them.
Everyone loves discussing specifics of mbufs and dev API's, and I have
no doubt that DPDK community can arrive at a consensus with regards to
mbuf format etc. in a timely manner, since everyone has a vested
interest in those covering their use cases. I have way less confidence
in us possibly having saner and more maintainable platform
initialization code, simply because any attempt to change those will
likely be met with "please keep all of the old stuff working", which
gets us right back to where we started.
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 10:15 ` Burakov, Anatoly
@ 2019-04-08 13:00 ` Ray Kinsella
2019-04-08 13:38 ` Burakov, Anatoly
0 siblings, 1 reply; 39+ messages in thread
From: Ray Kinsella @ 2019-04-08 13:00 UTC (permalink / raw)
To: Burakov, Anatoly, Thomas Monjalon
Cc: techboard, Bruce Richardson, dev, Kevin Traynor
On 08/04/2019 11:15, Burakov, Anatoly wrote:
> On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
>> On 07/04/2019 10:48, Thomas Monjalon wrote:
>>> 04/04/2019 16:07, Burakov, Anatoly:
>>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>> [SNIP]
[SNIP]
>
> My worry here is that some API's get more attention than others, but
> requirements for freezing the API/ABI are applicable to all of them.
>
> Everyone loves discussing specifics of mbufs and dev API's, and I have
> no doubt that DPDK community can arrive at a consensus with regards to
> mbuf format etc. in a timely manner, since everyone has a vested
> interest in those covering their use cases.
> I have way less confidence
> in us possibly having saner and more maintainable platform
> initialization code,
I think you are right, however its that same lack of enthusiasm that
would hamper a DPDK 1.0 specification. Similarly I can see a DPDK 1.0
specification effort becoming an endless process, as it would always be
low on DPDK contributors priority list.
Instead I recommend we set an API freeze date to focus peoples minds,
advertise what it means and contributors will respond and look after the
areas they care/are responsible for.
> simply because any attempt to change those will
> likely be met with "please keep all of the old stuff working", which
> gets us right back to where we started.
>
And to be honest that is a fair enough expectation from them, right?
To Stephen's if we need to break it, let's do it once more and then
never again.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 13:00 ` Ray Kinsella
@ 2019-04-08 13:38 ` Burakov, Anatoly
2019-04-08 13:58 ` David Marchand
2019-04-09 9:42 ` Ray Kinsella
0 siblings, 2 replies; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-08 13:38 UTC (permalink / raw)
To: Ray Kinsella, Thomas Monjalon
Cc: techboard, Bruce Richardson, dev, Kevin Traynor
On 08-Apr-19 2:00 PM, Ray Kinsella wrote:
>
>
> On 08/04/2019 11:15, Burakov, Anatoly wrote:
>> On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
>>> On 07/04/2019 10:48, Thomas Monjalon wrote:
>>>> 04/04/2019 16:07, Burakov, Anatoly:
>>>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>> [SNIP]
> [SNIP]
>>
>> My worry here is that some API's get more attention than others, but
>> requirements for freezing the API/ABI are applicable to all of them.
>>
>> Everyone loves discussing specifics of mbufs and dev API's, and I have
>> no doubt that DPDK community can arrive at a consensus with regards to
>> mbuf format etc. in a timely manner, since everyone has a vested
>> interest in those covering their use cases.
>> I have way less confidence
>> in us possibly having saner and more maintainable platform
>> initialization code,
>
> I think you are right, however its that same lack of enthusiasm that
> would hamper a DPDK 1.0 specification. Similarly I can see a DPDK 1.0
> specification effort becoming an endless process, as it would always be
> low on DPDK contributors priority list.
>
> Instead I recommend we set an API freeze date to focus peoples minds,
> advertise what it means and contributors will respond and look after the
> areas they care/are responsible for.
>
>> simply because any attempt to change those will
>> likely be met with "please keep all of the old stuff working", which
>> gets us right back to where we started.
>>
>
> And to be honest that is a fair enough expectation from them, right?
> To Stephen's if we need to break it, let's do it once more and then
> never again.
>
Perhaps i didn't phrase myself well enough. When i say "DPDK 1.0
specification" i don't mean there literally should be a document with a
formal specification :)
Rather, i'm thinking more along the lines of, let's take a look at what
we have, and think of ways we can reduce the amount of code and improve
things without causing major inconveniences to great majority of users.
As in, if we want to "break once more and then never again", then maybe
instead of small incremental fixes here and there, we could actually do
a more drastic change that keeps perhaps 90% users happy instead of
100%, but which reduces maintenance/validation effort from 100% down to 30%.
As a concrete proposal, my number one dream would be to see multiprocess
gone. I also recall desire for "DPDK to be more lightweight", and i
maintain that DPDK *cannot* be lightweight if we are to support
multiprocess - we can have one or the other, but not both. However,
realistically, i don't think dropping multiprocess is ever going to
happen - not only it is too entrenched in DPDK use cases, it is actually
quite useful despite its flaws.
However, what *could* be realistic is to trim down complexity in EAL
init and system code in general - to me (again, a concrete proposal!),
that would be dropping igb_uio and supporting upstream kernel modules
only (VFIO, maybe uio_pci_generic if we're pushing it?), dropping 32-bit
support and legacy mem modes, and perhaps bumping kernel version and
removing some compatibility code as well. This would remove potentially
thousands of lines of code and keep EAL cleaner and more maintainable:
right now, we have two different hotplug mechanisms (UIO and VFIO), lots
of different memory/interrupt modes, etc., for no reason that is readily
apparent to me.
So, as you can see, an effort to review and delineate what we want to
support would be necessary if we want to ease the maintenance burden for
either myself or anyone that inherits this code, as well as reducing the
number of moving parts and, as a consequence, validation effort and
amount of bugs. We can't simply do it in a random release "just
because", but if we were to "break once more and then never again",
perhaps this is something that could be considered.
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 13:38 ` Burakov, Anatoly
@ 2019-04-08 13:58 ` David Marchand
2019-04-08 14:02 ` Burakov, Anatoly
2019-04-09 9:42 ` Ray Kinsella
1 sibling, 1 reply; 39+ messages in thread
From: David Marchand @ 2019-04-08 13:58 UTC (permalink / raw)
To: Burakov, Anatoly
Cc: Ray Kinsella, Thomas Monjalon, techboard, Bruce Richardson, dev,
Kevin Traynor
On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> As a concrete proposal, my number one dream would be to see multiprocess
> gone. I also recall desire for "DPDK to be more lightweight", and i
> maintain that DPDK *cannot* be lightweight if we are to support
> multiprocess - we can have one or the other, but not both. However,
> realistically, i don't think dropping multiprocess is ever going to
> happen - not only it is too entrenched in DPDK use cases, it is actually
> quite useful despite its flaws.
>
Well, honestly, I'd like to hear about this.
What are the real usecases for multi process support?
Do we have even a single opensource project that uses it?
--
David Marchand
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 13:58 ` David Marchand
@ 2019-04-08 14:02 ` Burakov, Anatoly
2019-04-08 14:38 ` David Marchand
0 siblings, 1 reply; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-08 14:02 UTC (permalink / raw)
To: David Marchand
Cc: Ray Kinsella, Thomas Monjalon, techboard, Bruce Richardson, dev,
Kevin Traynor
On 08-Apr-19 2:58 PM, David Marchand wrote:
> On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> As a concrete proposal, my number one dream would be to see
> multiprocess
> gone. I also recall desire for "DPDK to be more lightweight", and i
> maintain that DPDK *cannot* be lightweight if we are to support
> multiprocess - we can have one or the other, but not both. However,
> realistically, i don't think dropping multiprocess is ever going to
> happen - not only it is too entrenched in DPDK use cases, it is
> actually
> quite useful despite its flaws.
>
>
> Well, honestly, I'd like to hear about this.
> What are the real usecases for multi process support?
> Do we have even a single opensource project that uses it?
>
I'm aware of a few closed source usages of multiprocess. I also think
current versions of collectd rely on secondary process (there's been a
Telemetry API added to avoid that, but AFAIK the support for Telemetry
is not upstream in collectd yet), and so do/would any dump-style
applications - in fact, we ourselves include one such application in our
codebase (pdump, proc-info, etc.).
>
> --
> David Marchand
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 14:02 ` Burakov, Anatoly
@ 2019-04-08 14:38 ` David Marchand
2019-04-08 15:13 ` Stephen Hemminger
` (2 more replies)
0 siblings, 3 replies; 39+ messages in thread
From: David Marchand @ 2019-04-08 14:38 UTC (permalink / raw)
To: Burakov, Anatoly
Cc: Ray Kinsella, Thomas Monjalon, techboard, Bruce Richardson, dev,
Kevin Traynor
On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> On 08-Apr-19 2:58 PM, David Marchand wrote:
> > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> >
> > As a concrete proposal, my number one dream would be to see
> > multiprocess
> > gone. I also recall desire for "DPDK to be more lightweight", and i
> > maintain that DPDK *cannot* be lightweight if we are to support
> > multiprocess - we can have one or the other, but not both. However,
> > realistically, i don't think dropping multiprocess is ever going to
> > happen - not only it is too entrenched in DPDK use cases, it is
> > actually
> > quite useful despite its flaws.
> >
> >
> > Well, honestly, I'd like to hear about this.
> > What are the real usecases for multi process support?
> > Do we have even a single opensource project that uses it?
> >
>
> I'm aware of a few closed source usages of multiprocess. I also think
> current versions of collectd rely on secondary process (there's been a
> Telemetry API added to avoid that, but AFAIK the support for Telemetry
> is not upstream in collectd yet), and so do/would any dump-style
> applications - in fact, we ourselves include one such application in our
> codebase (pdump, proc-info, etc.).
>
Sorry, I don't want to highjack this thread, I can start a separate thread
if people feel like it.
If we go with stabilisation, we must be careful that we want to support the
features.
So about multiprocess, again, in those closed source projects you know of,
what are the usecases?
For what we provide in dpdk pdump, proc-info, referring to oneself is not
that convincing to me as I don't use those tools.
I don't see what we could not achieve the same with a control thread
running in the dpdk process and handling commands.
It would be open to the outside via a more standard channel, like a UNIX
socket or something like this.
If we need to declare a dynamic channel, it can be constructed as an
extension of the existing standard channel: we can open something like a
POSIX shm and push things in it.
Was this explored ?
--
David Marchand
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 14:38 ` David Marchand
@ 2019-04-08 15:13 ` Stephen Hemminger
2019-04-08 15:49 ` Burakov, Anatoly
2019-04-08 15:50 ` Burakov, Anatoly
2 siblings, 0 replies; 39+ messages in thread
From: Stephen Hemminger @ 2019-04-08 15:13 UTC (permalink / raw)
To: David Marchand
Cc: Burakov, Anatoly, Ray Kinsella, Thomas Monjalon, techboard,
Bruce Richardson, dev, Kevin Traynor
On Mon, 8 Apr 2019 16:38:55 +0200
David Marchand <david.marchand@redhat.com> wrote:
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly <anatoly.burakov@intel.com>
> wrote:
>
> > On 08-Apr-19 2:58 PM, David Marchand wrote:
> > > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> > >
> > > As a concrete proposal, my number one dream would be to see
> > > multiprocess
> > > gone. I also recall desire for "DPDK to be more lightweight", and i
> > > maintain that DPDK *cannot* be lightweight if we are to support
> > > multiprocess - we can have one or the other, but not both. However,
> > > realistically, i don't think dropping multiprocess is ever going to
> > > happen - not only it is too entrenched in DPDK use cases, it is
> > > actually
> > > quite useful despite its flaws.
> > >
> > >
> > > Well, honestly, I'd like to hear about this.
> > > What are the real usecases for multi process support?
> > > Do we have even a single opensource project that uses it?
> > >
> >
> > I'm aware of a few closed source usages of multiprocess. I also think
> > current versions of collectd rely on secondary process (there's been a
> > Telemetry API added to avoid that, but AFAIK the support for Telemetry
> > is not upstream in collectd yet), and so do/would any dump-style
> > applications - in fact, we ourselves include one such application in our
> > codebase (pdump, proc-info, etc.).
> >
>
> Sorry, I don't want to highjack this thread, I can start a separate thread
> if people feel like it.
> If we go with stabilisation, we must be careful that we want to support the
> features.
>
> So about multiprocess, again, in those closed source projects you know of,
> what are the usecases?
>
> For what we provide in dpdk pdump, proc-info, referring to oneself is not
> that convincing to me as I don't use those tools.
>
> I don't see what we could not achieve the same with a control thread
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an
> extension of the existing standard channel: we can open something like a
> POSIX shm and push things in it.
> Was this explored ?
>
>
Yes, there are several closed source applications using multi-process.
But the problem with that is no one tests with all the drivers, api's and
configurations in DPDK.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 14:38 ` David Marchand
2019-04-08 15:13 ` Stephen Hemminger
@ 2019-04-08 15:49 ` Burakov, Anatoly
2019-04-10 8:35 ` David Marchand
2019-04-08 15:50 ` Burakov, Anatoly
2 siblings, 1 reply; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-08 15:49 UTC (permalink / raw)
To: David Marchand
Cc: Ray Kinsella, Thomas Monjalon, techboard, Bruce Richardson, dev,
Kevin Traynor
On 08-Apr-19 3:38 PM, David Marchand wrote:
>
>
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> On 08-Apr-19 2:58 PM, David Marchand wrote:
> > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
> <mailto:anatoly.burakov@intel.com
> <mailto:anatoly.burakov@intel.com>>> wrote:
> >
> > As a concrete proposal, my number one dream would be to see
> > multiprocess
> > gone. I also recall desire for "DPDK to be more lightweight",
> and i
> > maintain that DPDK *cannot* be lightweight if we are to support
> > multiprocess - we can have one or the other, but not both.
> However,
> > realistically, i don't think dropping multiprocess is ever
> going to
> > happen - not only it is too entrenched in DPDK use cases, it is
> > actually
> > quite useful despite its flaws.
> >
> >
> > Well, honestly, I'd like to hear about this.
> > What are the real usecases for multi process support?
> > Do we have even a single opensource project that uses it?
> >
>
> I'm aware of a few closed source usages of multiprocess. I also think
> current versions of collectd rely on secondary process (there's been a
> Telemetry API added to avoid that, but AFAIK the support for Telemetry
> is not upstream in collectd yet), and so do/would any dump-style
> applications - in fact, we ourselves include one such application in
> our
> codebase (pdump, proc-info, etc.).
>
>
> Sorry, I don't want to highjack this thread, I can start a separate
> thread if people feel like it.
> If we go with stabilisation, we must be careful that we want to support
> the features.
>
> So about multiprocess, again, in those closed source projects you know
> of, what are the usecases?
>
> For what we provide in dpdk pdump, proc-info, referring to oneself is
> not that convincing to me as I don't use those tools.
>
> I don't see what we could not achieve the same with a control thread
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an
> extension of the existing standard channel: we can open something like a
> POSIX shm and push things in it.
> Was this explored ?
There are certainly things that we can do that can make some aspects of
multiprocess redundant. For example, for any kind of collectd-like
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably
provide a better and more maintainable way of doing things.
Our multiprocess also makes it easier to write pipeline/load-balancing
type applications. To see an example, look at our
multiprocess/client-server example. This is demonstrating how, instead
of writing one big monolithic application, one could instead write a
number of smaller applications each doing their thing. It is of course
possible to do the same without multiprocess, as evidenced by our sample
applications such as load-balancer, distributor, ip-pipeline etc., but
it is arguably easier to implement *real* applications that way due to
separation of concerns and more focused codebase.
However, there are two use cases that i can think of that are either
hard or outright not possible without our multiprocess API's. The first
one is dumping functionality. For example, dpdk_proc_info can display
info from a currently-running or defunct process - list its
memzones/mempools/etc. - basically, everything there is to know about
the shared memory can be known that way. While this isn't a "real" use
case, it is useful for debugging.
More importantly, our multiprocess model provides resilience. In an
event of a crash, the entire application is not brought down - instead,
only the crashed process goes down. It's not /perfect/ resilience, of
course, and there are caveats (memory leaking, locks, etc.), but you do
get /some/ resilience that way - your process went down, you spin
another secondary and you're back up and running again.
The above described scenario is how most people (that i know of) appear
to be using multiprocess - some kind of "crash-resilient"
load-balancing/pipelining app.
>
>
> --
> David Marchand
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 15:49 ` Burakov, Anatoly
@ 2019-04-10 8:35 ` David Marchand
0 siblings, 0 replies; 39+ messages in thread
From: David Marchand @ 2019-04-10 8:35 UTC (permalink / raw)
To: Burakov, Anatoly
Cc: Ray Kinsella, Thomas Monjalon, techboard, Bruce Richardson, dev,
Kevin Traynor
On Mon, Apr 8, 2019 at 5:52 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
However, there are two use cases that i can think of that are either
> hard or outright not possible without our multiprocess API's. The first
> one is dumping functionality. For example, dpdk_proc_info can display
> info from a currently-running or defunct process - list its
> memzones/mempools/etc. - basically, everything there is to know about
> the shared memory can be known that way. While this isn't a "real" use
> case, it is useful for debugging.
>
On the principle (there might be some corner cases I don't have an idea of)
attaching an existing process is something that can be done with PTRACE_*.
As for dumping post mortem, again, this could be done without dpdk assist.
The only thing that should come from dpdk is the knowledge of where and how
to dump the objects.
> More importantly, our multiprocess model provides resilience. In an
> event of a crash, the entire application is not brought down - instead,
> only the crashed process goes down. It's not /perfect/ resilience, of
> course, and there are caveats (memory leaking, locks, etc.), but you do
> get /some/ resilience that way - your process went down, you spin
> another secondary and you're back up and running again.
>
Never witnessed a recover, and yes I would suspect quite some funny
situations like locks, dirty memory etc...
Okay, thanks for the inputs.
--
David Marchand
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 14:38 ` David Marchand
2019-04-08 15:13 ` Stephen Hemminger
2019-04-08 15:49 ` Burakov, Anatoly
@ 2019-04-08 15:50 ` Burakov, Anatoly
2 siblings, 0 replies; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-08 15:50 UTC (permalink / raw)
To: David Marchand
Cc: Ray Kinsella, Thomas Monjalon, techboard, Bruce Richardson, dev,
Kevin Traynor
On 08-Apr-19 3:38 PM, David Marchand wrote:
>
>
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> On 08-Apr-19 2:58 PM, David Marchand wrote:
> > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
> <mailto:anatoly.burakov@intel.com
> <mailto:anatoly.burakov@intel.com>>> wrote:
> >
> > As a concrete proposal, my number one dream would be to see
> > multiprocess
> > gone. I also recall desire for "DPDK to be more lightweight",
> and i
> > maintain that DPDK *cannot* be lightweight if we are to support
> > multiprocess - we can have one or the other, but not both.
> However,
> > realistically, i don't think dropping multiprocess is ever
> going to
> > happen - not only it is too entrenched in DPDK use cases, it is
> > actually
> > quite useful despite its flaws.
> >
> >
> > Well, honestly, I'd like to hear about this.
> > What are the real usecases for multi process support?
> > Do we have even a single opensource project that uses it?
> >
>
> I'm aware of a few closed source usages of multiprocess. I also think
> current versions of collectd rely on secondary process (there's been a
> Telemetry API added to avoid that, but AFAIK the support for Telemetry
> is not upstream in collectd yet), and so do/would any dump-style
> applications - in fact, we ourselves include one such application in
> our
> codebase (pdump, proc-info, etc.).
>
>
> Sorry, I don't want to highjack this thread, I can start a separate
> thread if people feel like it.
> If we go with stabilisation, we must be careful that we want to support
> the features.
>
> So about multiprocess, again, in those closed source projects you know
> of, what are the usecases?
>
> For what we provide in dpdk pdump, proc-info, referring to oneself is
> not that convincing to me as I don't use those tools.
>
> I don't see what we could not achieve the same with a control thread
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an
> extension of the existing standard channel: we can open something like a
> POSIX shm and push things in it.
> Was this explored ?
There are certainly things that we can do that can make some aspects of
multiprocess redundant. For example, for any kind of collectd-like
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably
provide a better and more maintainable way of doing things.
Our multiprocess also makes it easier to write pipeline/load-balancing
type applications. To see an example, look at our
multiprocess/client-server example. This is demonstrating how, instead
of writing one big monolithic application, one could instead write a
number of smaller applications each doing their thing. It is of course
possible to do the same without multiprocess, as evidenced by our sample
applications such as load-balancer, distributor, ip-pipeline etc., but
it is arguably easier to implement *real* applications that way due to
separation of concerns and more focused codebase.
However, there are two use cases that i can think of that are either
hard or outright not possible without our multiprocess API's. The first
one is dumping functionality. For example, dpdk_proc_info can display
info from a currently-running or defunct process - list its
memzones/mempools/etc. - basically, everything there is to know about
the shared memory can be known that way. While this isn't a "real" use
case, it is useful for debugging.
More importantly, our multiprocess model provides resilience. In an
event of a crash, the entire application is not brought down - instead,
only the crashed process goes down. It's not /perfect/ resilience, of
course, and there are caveats (memory leaking, locks, etc.), but you do
get /some/ resilience that way - your process went down, you spin
another secondary and you're back up and running again.
The above described scenario is how most people (that i know of) appear
to be using multiprocess - some kind of "crash-resilient"
load-balancing/pipelining app.
>
>
> --
> David Marchand
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 13:38 ` Burakov, Anatoly
2019-04-08 13:58 ` David Marchand
@ 2019-04-09 9:42 ` Ray Kinsella
1 sibling, 0 replies; 39+ messages in thread
From: Ray Kinsella @ 2019-04-09 9:42 UTC (permalink / raw)
To: Burakov, Anatoly, Thomas Monjalon
Cc: techboard, Bruce Richardson, dev, Kevin Traynor
On 08/04/2019 14:38, Burakov, Anatoly wrote:
> On 08-Apr-19 2:00 PM, Ray Kinsella wrote:
>>
>>
>> On 08/04/2019 11:15, Burakov, Anatoly wrote:
>>> On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
>>>> On 07/04/2019 10:48, Thomas Monjalon wrote:
>>>>> 04/04/2019 16:07, Burakov, Anatoly:
>>>>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>>> [SNIP]
>> [SNIP]
[SNIP]
>>
>
> Perhaps i didn't phrase myself well enough. When i say "DPDK 1.0
> specification" i don't mean there literally should be a document with a
> formal specification :)
When I hear the word specification I think encyclopedia-like stacks of
paper.
>
> Rather, i'm thinking more along the lines of, let's take a look at what
> we have, and think of ways we can reduce the amount of code and improve
> things without causing major inconveniences to great majority of users.
> As in, if we want to "break once more and then never again", then maybe
> instead of small incremental fixes here and there, we could actually do
> a more drastic change that keeps perhaps 90% users happy instead of
> 100%, but which reduces maintenance/validation effort from 100% down to
> 30%.
Agreed - all I would propose on top of this is that we give ourselves a
real deadline.
The DPDK API has been iterating since 2012 and while we can spare a few
more months to do tidy-up and get it right, we _do_ need to draw a line
in the sand at the same time.
>
> As a concrete proposal, my number one dream would be to see multiprocess
> gone. I also recall desire for "DPDK to be more lightweight", and i
> maintain that DPDK *cannot* be lightweight if we are to support
> multiprocess - we can have one or the other, but not both. However,
> realistically, i don't think dropping multiprocess is ever going to
> happen - not only it is too entrenched in DPDK use cases, it is actually
> quite useful despite its flaws.
>
> However, what *could* be realistic is to trim down complexity in EAL
> init and system code in general - to me (again, a concrete proposal!),
> that would be dropping igb_uio and supporting upstream kernel modules
> only (VFIO, maybe uio_pci_generic if we're pushing it?), dropping 32-bit
> support and legacy mem modes, and perhaps bumping kernel version and
> removing some compatibility code as well. This would remove potentially
> thousands of lines of code and keep EAL cleaner and more maintainable:
> right now, we have two different hotplug mechanisms (UIO and VFIO), lots
> of different memory/interrupt modes, etc., for no reason that is readily
> apparent to me.
>
> So, as you can see, an effort to review and delineate what we want to
> support would be necessary if we want to ease the maintenance burden for
> either myself or anyone that inherits this code, as well as reducing the
> number of moving parts and, as a consequence, validation effort and
> amount of bugs. We can't simply do it in a random release "just
> because", but if we were to "break once more and then never again",
> perhaps this is something that could be considered.
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-08 9:04 ` Ray Kinsella
2019-04-08 10:15 ` Burakov, Anatoly
@ 2019-04-14 0:42 ` Neil Horman
2019-04-15 9:10 ` Bruce Richardson
1 sibling, 1 reply; 39+ messages in thread
From: Neil Horman @ 2019-04-14 0:42 UTC (permalink / raw)
To: Ray Kinsella; +Cc: dev
On Mon, Apr 08, 2019 at 10:04:21AM +0100, Ray Kinsella wrote:
> On 07/04/2019 10:48, Thomas Monjalon wrote:
> > 04/04/2019 16:07, Burakov, Anatoly:
> >> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> >>> On 04/04/2019 11:54, Bruce Richardson wrote:
> >>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> [SNIP]
> >> So, if we are to cement our core API - we have to make a concrete effort
> >> to specify what goes and what stays, if we want it to be maintainable.
> >> The DPDK 1.0 specification, if you will :)
> >
> > "DPDK 1.0 specification", that's a great project name :-)
> >
>
> Honestly - I would say that I am nervous of this.
>
> The definition of a DPDK 1.0 specification as a gate to API stability,
> feels like a "great plan tomorrow" instead of a "good plan" today. I
> think that getting people to dedicate time to such a specification might
> prove problematic and I could see this effort being very time consuming.
> It might never get completed.
>
> My preference would be to instead adopt a well-publicised community
> timeline for adopting more conservative API maintenance rules.
>
> Perhaps we could give ourselves as a community a time-limited window in
> which to address concerns around the API before they become hardened -
> perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
> to 9 months.
>
> We then would know the timeline when niggles like exposure of internal
> structures and mbuf structure needed to be sorted by and could
> prioritize accordingly.
>
> Ray K
I'm hesitant to say this, because I'm not usually a fan of throwing up
barricades to progress, but might some level of CI integration be useful here?
Part of the problem, as I've seen it (and I think you've noted previously in
this thread), is that ABI stability just hasn't been a priority, and not
something that developers look at when making changes, nor when reviewers review
patches. When I wrote the early ABI checking tools for DPDK, while the reaction
was generally positive (I think), the results were informational, and treated as
such (something to take note of perhaps, but something that could be ignored if
there were more pressing issues). Perhaps a concrete step might be to run the
ABI checker during a CI run on every commit, and block acceptance of a patch if
it modifies the ABI. That would at least put a procedural break in ABI
modification without clear approval from the board.
Neil
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-14 0:42 ` Neil Horman
@ 2019-04-15 9:10 ` Bruce Richardson
0 siblings, 0 replies; 39+ messages in thread
From: Bruce Richardson @ 2019-04-15 9:10 UTC (permalink / raw)
To: Neil Horman; +Cc: Ray Kinsella, dev
On Sat, Apr 13, 2019 at 08:42:02PM -0400, Neil Horman wrote:
> On Mon, Apr 08, 2019 at 10:04:21AM +0100, Ray Kinsella wrote:
> > On 07/04/2019 10:48, Thomas Monjalon wrote:
> > > 04/04/2019 16:07, Burakov, Anatoly:
> > >> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> > >>> On 04/04/2019 11:54, Bruce Richardson wrote:
> > >>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> > >>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> > [SNIP]
> > >> So, if we are to cement our core API - we have to make a concrete effort
> > >> to specify what goes and what stays, if we want it to be maintainable.
> > >> The DPDK 1.0 specification, if you will :)
> > >
> > > "DPDK 1.0 specification", that's a great project name :-)
> > >
> >
> > Honestly - I would say that I am nervous of this.
> >
> > The definition of a DPDK 1.0 specification as a gate to API stability,
> > feels like a "great plan tomorrow" instead of a "good plan" today. I
> > think that getting people to dedicate time to such a specification might
> > prove problematic and I could see this effort being very time consuming.
> > It might never get completed.
> >
> > My preference would be to instead adopt a well-publicised community
> > timeline for adopting more conservative API maintenance rules.
> >
> > Perhaps we could give ourselves as a community a time-limited window in
> > which to address concerns around the API before they become hardened -
> > perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
> > to 9 months.
> >
> > We then would know the timeline when niggles like exposure of internal
> > structures and mbuf structure needed to be sorted by and could
> > prioritize accordingly.
> >
> > Ray K
>
> I'm hesitant to say this, because I'm not usually a fan of throwing up
> barricades to progress, but might some level of CI integration be useful here?
>
> Part of the problem, as I've seen it (and I think you've noted previously in
> this thread), is that ABI stability just hasn't been a priority, and not
> something that developers look at when making changes, nor when reviewers review
> patches. When I wrote the early ABI checking tools for DPDK, while the reaction
> was generally positive (I think), the results were informational, and treated as
> such (something to take note of perhaps, but something that could be ignored if
> there were more pressing issues). Perhaps a concrete step might be to run the
> ABI checker during a CI run on every commit, and block acceptance of a patch if
> it modifies the ABI. That would at least put a procedural break in ABI
> modification without clear approval from the board.
>
No objections to that here. Sounds a reasonable suggestion.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 10:54 ` [dpdk-techboard] " Bruce Richardson
2019-04-04 12:02 ` Luca Boccassi
2019-04-04 12:52 ` Ray Kinsella
@ 2019-04-04 15:51 ` Stephen Hemminger
2019-04-04 16:37 ` Burakov, Anatoly
2019-04-04 16:56 ` Kevin Traynor
3 siblings, 1 reply; 39+ messages in thread
From: Stephen Hemminger @ 2019-04-04 15:51 UTC (permalink / raw)
To: Bruce Richardson
Cc: Burakov, Anatoly, Ray Kinsella, dev, Kevin Traynor, techboard
On Thu, 4 Apr 2019 11:54:47 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should reduce
> the scope for ABI breaks.
> 2. Once done, I think we should commit to having an ABI break only in the
> rarest of circumstances, and only with very large justification. I want us
> to get to the point where DPDK releases can immediately be picked up by all
> linux distros and rolled out because they are ABI compatible.
I would also like to propose "you get one ABI break" which means each
API/ABI change must hide more infrastructure than the last. This is
the "fool me once, ..." saying in API's.
For example,
the memory rework it would have been good if the structure of mempools etc
were hidden inside EAL and not exposed. but as usual hindsight is 20/20
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 15:51 ` Stephen Hemminger
@ 2019-04-04 16:37 ` Burakov, Anatoly
0 siblings, 0 replies; 39+ messages in thread
From: Burakov, Anatoly @ 2019-04-04 16:37 UTC (permalink / raw)
To: Stephen Hemminger, Bruce Richardson
Cc: Ray Kinsella, dev, Kevin Traynor, techboard
On 04-Apr-19 4:51 PM, Stephen Hemminger wrote:
> On Thu, 4 Apr 2019 11:54:47 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
>
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should reduce
>> the scope for ABI breaks.
>> 2. Once done, I think we should commit to having an ABI break only in the
>> rarest of circumstances, and only with very large justification. I want us
>> to get to the point where DPDK releases can immediately be picked up by all
>> linux distros and rolled out because they are ABI compatible.
>
> I would also like to propose "you get one ABI break" which means each
> API/ABI change must hide more infrastructure than the last. This is
> the "fool me once, ..." saying in API's.
>
> For example,
> the memory rework it would have been good if the structure of mempools etc
> were hidden inside EAL and not exposed. but as usual hindsight is 20/20
>
Mempools is not part of "memory rework" - it's a separate library built
on top of EAL's memory subsystem :) When i talk about "memory API's", i
mean memzone/malloc and friends, not mempool.
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 10:54 ` [dpdk-techboard] " Bruce Richardson
` (2 preceding siblings ...)
2019-04-04 15:51 ` Stephen Hemminger
@ 2019-04-04 16:56 ` Kevin Traynor
2019-04-04 19:08 ` Wiles, Keith
3 siblings, 1 reply; 39+ messages in thread
From: Kevin Traynor @ 2019-04-04 16:56 UTC (permalink / raw)
To: Bruce Richardson, Burakov, Anatoly; +Cc: Ray Kinsella, dev, techboard
On 04/04/2019 11:54, Bruce Richardson wrote:
<snip>
>
> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should reduce
> the scope for ABI breaks.
> 2. Once done, I think we should commit to having an ABI break only in the
> rarest of circumstances, and only with very large justification. I want us
> to get to the point where DPDK releases can immediately be picked up by all
> linux distros and rolled out because they are ABI compatible.
>
Maybe techboard should explicitly approve ABI breaks and new APIs (or
APIs at transition from experimental to core). Just as a way to get more
eyeballs and scrutiny on them.
> I'm not sure I like the idea of planned ABI break releases - that strikes
> me as a plan where we end up with the same number of ABI breaks as before,
> just balled into one release.
>
> Question for Kevin, Luca and others who look at distro-packaging: is it the
> case that each distro will only ship one version of DPDK, or is it possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>
It would probably double validation and maintenance, so it would require
a lot of extra effort.
>
> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory layout or
> similar.
>
> Regards,
> /Bruce
>
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 16:56 ` Kevin Traynor
@ 2019-04-04 19:08 ` Wiles, Keith
2019-04-04 20:13 ` Kevin Traynor
2019-04-05 13:29 ` Ray Kinsella
0 siblings, 2 replies; 39+ messages in thread
From: Wiles, Keith @ 2019-04-04 19:08 UTC (permalink / raw)
To: Kevin Traynor
Cc: Richardson, Bruce, Burakov, Anatoly, Ray Kinsella, dpdk-dev, techboard
> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>
> On 04/04/2019 11:54, Bruce Richardson wrote:
> <snip>
>
>>
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should reduce
>> the scope for ABI breaks.
>> 2. Once done, I think we should commit to having an ABI break only in the
>> rarest of circumstances, and only with very large justification. I want us
>> to get to the point where DPDK releases can immediately be picked up by all
>> linux distros and rolled out because they are ABI compatible.
>>
>
> Maybe techboard should explicitly approve ABI breaks and new APIs (or
> APIs at transition from experimental to core). Just as a way to get more
> eyeballs and scrutiny on them.
ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
>
>> I'm not sure I like the idea of planned ABI break releases - that strikes
>> me as a plan where we end up with the same number of ABI breaks as before,
>> just balled into one release.
>>
>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>> case that each distro will only ship one version of DPDK, or is it possible
>> that if we have ABI breaks, a distro will provide two copies of DPDK
>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>
>
> It would probably double validation and maintenance, so it would require
> a lot of extra effort.
>
>>
>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>> ABI breaks, and having ABI breaks as a major event reserved only for
>> massive rework changes, such as major mbuf changes, or new memory layout or
>> similar.
>>
>> Regards,
>> /Bruce
>>
>
Regards,
Keith
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 19:08 ` Wiles, Keith
@ 2019-04-04 20:13 ` Kevin Traynor
2019-04-05 13:30 ` [dpdk-dev] " Ray Kinsella
2019-04-05 13:29 ` Ray Kinsella
1 sibling, 1 reply; 39+ messages in thread
From: Kevin Traynor @ 2019-04-04 20:13 UTC (permalink / raw)
To: Wiles, Keith
Cc: Richardson, Bruce, Burakov, Anatoly, Ray Kinsella, dpdk-dev, techboard
On 04/04/2019 20:08, Wiles, Keith wrote:
>
>
>> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>>
>> On 04/04/2019 11:54, Bruce Richardson wrote:
>> <snip>
>>
>>>
>>> My thoughts on the matter are:
>>> 1. I think we really need to do work to start hiding more of our data
>>> structures - like what Stephen's latest RFC does. This hiding should reduce
>>> the scope for ABI breaks.
>>> 2. Once done, I think we should commit to having an ABI break only in the
>>> rarest of circumstances, and only with very large justification. I want us
>>> to get to the point where DPDK releases can immediately be picked up by all
>>> linux distros and rolled out because they are ABI compatible.
>>>
>>
>> Maybe techboard should explicitly approve ABI breaks and new APIs (or
>> APIs at transition from experimental to core). Just as a way to get more
>> eyeballs and scrutiny on them.
>
> ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
>>
We'll only find out if they are bad when they need ABI breaks later :-)
My point is a good way to avoid future ABI breaks is to have more
reviews on the APIs in the first place. Techboard approval might be one
way, or 3 acks or something else.
>>> I'm not sure I like the idea of planned ABI break releases - that strikes
>>> me as a plan where we end up with the same number of ABI breaks as before,
>>> just balled into one release.
>>>
>>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>>> case that each distro will only ship one version of DPDK, or is it possible
>>> that if we have ABI breaks, a distro will provide two copies of DPDK
>>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>>
>>
>> It would probably double validation and maintenance, so it would require
>> a lot of extra effort.
>>
>>>
>>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>>> ABI breaks, and having ABI breaks as a major event reserved only for
>>> massive rework changes, such as major mbuf changes, or new memory layout or
>>> similar.
>>>
>>> Regards,
>>> /Bruce
>>>
>>
>
> Regards,
> Keith
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 20:13 ` Kevin Traynor
@ 2019-04-05 13:30 ` Ray Kinsella
0 siblings, 0 replies; 39+ messages in thread
From: Ray Kinsella @ 2019-04-05 13:30 UTC (permalink / raw)
To: Kevin Traynor, Wiles, Keith
Cc: Richardson, Bruce, Burakov, Anatoly, dpdk-dev, techboard
On 04/04/2019 21:13, Kevin Traynor wrote:
> On 04/04/2019 20:08, Wiles, Keith wrote:
>>
>>
>>> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>>>
>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>> <snip>
>>>
>>>>
>>>> My thoughts on the matter are:
>>>> 1. I think we really need to do work to start hiding more of our data
>>>> structures - like what Stephen's latest RFC does. This hiding should reduce
>>>> the scope for ABI breaks.
>>>> 2. Once done, I think we should commit to having an ABI break only in the
>>>> rarest of circumstances, and only with very large justification. I want us
>>>> to get to the point where DPDK releases can immediately be picked up by all
>>>> linux distros and rolled out because they are ABI compatible.
>>>>
>>>
>>> Maybe techboard should explicitly approve ABI breaks and new APIs (or
>>> APIs at transition from experimental to core). Just as a way to get more
>>> eyeballs and scrutiny on them.
>>
>> ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
>>>
>
> We'll only find out if they are bad when they need ABI breaks later :-)
>
> My point is a good way to avoid future ABI breaks is to have more
> reviews on the APIs in the first place. Techboard approval might be one
> way, or 3 acks or something else.
+1 on this ... an API break should invite a higher level of scrutiny.
>
>>>> I'm not sure I like the idea of planned ABI break releases - that strikes
>>>> me as a plan where we end up with the same number of ABI breaks as before,
>>>> just balled into one release.
>>>>
>>>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>>>> case that each distro will only ship one version of DPDK, or is it possible
>>>> that if we have ABI breaks, a distro will provide two copies of DPDK
>>>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>>>
>>>
>>> It would probably double validation and maintenance, so it would require
>>> a lot of extra effort.
>>>
>>>>
>>>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>>>> ABI breaks, and having ABI breaks as a major event reserved only for
>>>> massive rework changes, such as major mbuf changes, or new memory layout or
>>>> similar.
>>>>
>>>> Regards,
>>>> /Bruce
>>>>
>>>
>>
>> Regards,
>> Keith
>>
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
2019-04-04 19:08 ` Wiles, Keith
2019-04-04 20:13 ` Kevin Traynor
@ 2019-04-05 13:29 ` Ray Kinsella
1 sibling, 0 replies; 39+ messages in thread
From: Ray Kinsella @ 2019-04-05 13:29 UTC (permalink / raw)
To: Wiles, Keith, Kevin Traynor
Cc: Richardson, Bruce, Burakov, Anatoly, dpdk-dev, techboard
On 04/04/2019 20:08, Wiles, Keith wrote:
>
>
>> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>>
>> On 04/04/2019 11:54, Bruce Richardson wrote:
>> <snip>
> ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
New APIs will be experimental in any case, as you say they are less of
problem.
I agree, if we can make a change and preserve API compatibility with
versioning then everyone is happy.
Changes only need be referred to the higher power in case on absolutely
breakage - _but_ these need to become as rare as hens teeth.
>>
>>> I'm not sure I like the idea of planned ABI break releases - that strikes
>>> me as a plan where we end up with the same number of ABI breaks as before,
>>> just balled into one release.
>>>
>>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>>> case that each distro will only ship one version of DPDK, or is it possible
>>> that if we have ABI breaks, a distro will provide two copies of DPDK
>>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>>
>>
>> It would probably double validation and maintenance, so it would require
>> a lot of extra effort.
>>
>>>
>>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>>> ABI breaks, and having ABI breaks as a major event reserved only for
>>> massive rework changes, such as major mbuf changes, or new memory layout or
>>> similar.
>>>
>>> Regards,
>>> /Bruce
>>>
>>
>
> Regards,
> Keith
>
^ permalink raw reply [flat|nested] 39+ messages in thread