DPDK ABI/API Stability

DPDK ABI/API Stability

[Ray Kinsella]

Hi folks,

Recently I started a discussion with the DPDK Technical Board on DPDK
ABI/API stability. This was born out informal feedback I had received
from a number of users of DPDK about ABI churn. In turn this feedback
then prompted an ABI analysis of DPDK using tools from abi-laboratory.

https://abi-laboratory.pro/index.php?view=timeline&l=dpdk

I guess the short story is that DPDK ABI hasn't really settled down as
the project has matured. If you take a look at the “Backward Compat.”
column which measures ABI compatibility compared to the previous
releases, you will see significant churn in the ABI over successive
releases since v16.04.

Now compare DPDK to GStreamer as an example of a very mature project
with a similar intent, a framework for building applications, and which
enjoys a very stable API.

https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer

The DPDK ABI churn has the following affects for users:-

1. The churn obliges users of DPDK to commit to a constant
re-integration and re-validation effort for new versions of DPDK. This
effort from their perspective may not add value to their consuming
project, particular if they are only updating to "stay current".
2. The churn encourages users of DPDK to slip versions, putting off
reintegration to later, building up technical debt and causing their
projects to miss support for new hardware or features.
3. It makes DPDK different to almost every other system library and
framework that an operating systems might ship. This makes DPDK trickier
to dynamically link against, package and maintain for OS maintainers.

In order to address this issue, I have put together the minimal set of
concrete proposals below for discussion at the Technical Board next
Wednesday.

I wanted to share this, as these might not yet be the right proposals,
however I am putting them out there for feedback to start the discussion.

Thanks,

Ray K


Experimental API
1.	APIs designated as experimental are not considered part of the ABI
and may change without warning at any time.
2.	APIs designated as experimental must be marked depreciated for a
least one quarterly release before removal.
3.	APIs designated as experimental will no longer automatically graduate
to core after one release, they may stay experimental until their author
and the maintainer agree that graduation is appropriate.

Core API (non-experimental API)
4.	APIs designated as core must be depreciated for a least two years
before removal, to facilitate the continued compatibility with LTS
releases. A final removal notice will be published to the DPDK Mailing
List, and if there are no strong objections only then an API may be
removed.
5.	APIs designated as core may be changed as follows:-
5.a	The change proposer must demonstrated that the change has a
supporting use case and could not be achieved in any other way.
5.b	ABI version compatibility must be retained, as described below.

Shared Libraries
6.	DPDK will move to shared libraries & dynamic linking by default, to
accommodate greater use of ABI versioning by DPDK consumers.

ABI Versioning
7.	New quarterly releases of DPDK will remain ABI compatible with the
most recent DPDK LTS release.
(e.g. DPDK 19.08 will remain ABI compatible with DPDK LTS 18.11).
8.	New DPDK LTS releases will remain ABI compatible with the previous
two DPDK LTS releases.
(e.g. DPDK 20.11 will be ABI compatible with DPDK 19.11 and DPDK 18.11,
DPDK 21.11 will be ABI compatible with DPDK 20.11 and DPDK 19.11 etc)
8. & 9. will be achieved with ABI symbol versioning.

Re: DPDK ABI/API Stability

[Luca Boccassi]

On Wed, 2019-04-03 at 16:42 +0100, Ray Kinsella wrote:
> Hi folks,
> 
> Recently I started a discussion with the DPDK Technical Board on DPDK
> ABI/API stability. This was born out informal feedback I had received
> from a number of users of DPDK about ABI churn. In turn this feedback
> then prompted an ABI analysis of DPDK using tools from abi-
> laboratory.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> 
> 
> I guess the short story is that DPDK ABI hasn't really settled down
> as
> the project has matured. If you take a look at the “Backward Compat.”
> column which measures ABI compatibility compared to the previous
> releases, you will see significant churn in the ABI over successive
> releases since v16.04.
> 
> Now compare DPDK to GStreamer as an example of a very mature project
> with a similar intent, a framework for building applications, and
> which
> enjoys a very stable API.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> 
> 
> The DPDK ABI churn has the following affects for users:-
> 
> 1. The churn obliges users of DPDK to commit to a constant
> re-integration and re-validation effort for new versions of DPDK.
> This
> effort from their perspective may not add value to their consuming
> project, particular if they are only updating to "stay current".
> 2. The churn encourages users of DPDK to slip versions, putting off
> reintegration to later, building up technical debt and causing their
> projects to miss support for new hardware or features.
> 3. It makes DPDK different to almost every other system library and
> framework that an operating systems might ship. This makes DPDK
> trickier
> to dynamically link against, package and maintain for OS maintainers.
> 
> In order to address this issue, I have put together the minimal set
> of
> concrete proposals below for discussion at the Technical Board next
> Wednesday.
> 
> I wanted to share this, as these might not yet be the right
> proposals,
> however I am putting them out there for feedback to start the
> discussion.
> 
> Thanks,
> 
> Ray K

Hello Ray,

It will not come as a surprise to anybody when I say I fully agree with
your analysis :-)

One of the consequences is that this has a tendency to lead to
vendorization - given distros have to move on, and moving on with
application is a lot of work as you pointed out, the easiest solution
is to fork and embed.

If we can't change this, I think at the very least we should be more
explicit in the documentation about how the API is in flux. At least
project managers would be able to know up front that there is a higher-
than-normal long-term opex attached to a DPDK application, even when
all other development stops.

A couple of comments inline below.

> Experimental API
> 1.	APIs designated as experimental are not considered part of the
> ABI
> and may change without warning at any time.
> 2.	APIs designated as experimental must be marked depreciated for
> a
> least one quarterly release before removal.
> 3.	APIs designated as experimental will no longer automatically
> graduate
> to core after one release, they may stay experimental until their
> author
> and the maintainer agree that graduation is appropriate.

>From my experience in other projects, if a symbol is available in a
shared library, it will be used. So one thing that's missing from the
current system, which is already good and very helpful, is being able
to completely disable the experimental APIs.
In the past I looked into stopping to export those symbols but the
issue is intra-project dependencies - this can only be done if, for
example, PMDs are not allowed to use an experimental RTE api, unless
it's wrapped in an experimental ifdef (or similar mechanism). This of
course would be a major pain for everybody involved.

> Core API (non-experimental API)
> 4.	APIs designated as core must be depreciated for a least two
> years
> before removal, to facilitate the continued compatibility with LTS
> releases. A final removal notice will be published to the DPDK
> Mailing
> List, and if there are no strong objections only then an API may be
> removed.
> 5.	APIs designated as core may be changed as follows:-
> 5.a	The change proposer must demonstrated that the change has a
> supporting use case and could not be achieved in any other way.
> 5.b	ABI version compatibility must be retained, as described below.
> 
> Shared Libraries
> 6.	DPDK will move to shared libraries & dynamic linking by
> default, to
> accommodate greater use of ABI versioning by DPDK consumers.

The switch to Meson will help here, as it's the default.

> ABI Versioning
> 7.	New quarterly releases of DPDK will remain ABI compatible with
> the
> most recent DPDK LTS release.
> (e.g. DPDK 19.08 will remain ABI compatible with DPDK LTS 18.11).
> 8.	New DPDK LTS releases will remain ABI compatible with the
> previous
> two DPDK LTS releases.
> (e.g. DPDK 20.11 will be ABI compatible with DPDK 19.11 and DPDK
> 18.11,
> DPDK 21.11 will be ABI compatible with DPDK 20.11 and DPDK 19.11 etc)
> 8. & 9. will be achieved with ABI symbol versioning.

There is also an additional problem with the way we do ABI versioning,
we talked about it many times - ABI bumps need to be "sticky" and
propagate. Eg: if libfoo goes from 1 to 2, all libraries linking to foo
also need a bump. We don't do this, which made necessary the
introduction of the "major abi revision" option, which essentially
makes all the ABI versions match the release version, and which we use
in Debian/Ubuntu.

IMHO we should do a mix of the two - move to always use major version,
but of the LTS as you suggest. So all ABI revisions will be 19.11 until
20.11 is out, and so on.

-- 
Kind regards,
Luca Boccassi

Re: DPDK ABI/API Stability

[Burakov, Anatoly]

On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> Hi folks,
> 
> Recently I started a discussion with the DPDK Technical Board on DPDK
> ABI/API stability. This was born out informal feedback I had received
> from a number of users of DPDK about ABI churn. In turn this feedback
> then prompted an ABI analysis of DPDK using tools from abi-laboratory.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> 
> I guess the short story is that DPDK ABI hasn't really settled down as
> the project has matured. If you take a look at the “Backward Compat.”
> column which measures ABI compatibility compared to the previous
> releases, you will see significant churn in the ABI over successive
> releases since v16.04.
> 
> Now compare DPDK to GStreamer as an example of a very mature project
> with a similar intent, a framework for building applications, and which
> enjoys a very stable API.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> 
> The DPDK ABI churn has the following affects for users:-
> 
> 1. The churn obliges users of DPDK to commit to a constant
> re-integration and re-validation effort for new versions of DPDK. This
> effort from their perspective may not add value to their consuming
> project, particular if they are only updating to "stay current".
> 2. The churn encourages users of DPDK to slip versions, putting off
> reintegration to later, building up technical debt and causing their
> projects to miss support for new hardware or features.
> 3. It makes DPDK different to almost every other system library and
> framework that an operating systems might ship. This makes DPDK trickier
> to dynamically link against, package and maintain for OS maintainers.
> 
> In order to address this issue, I have put together the minimal set of
> concrete proposals below for discussion at the Technical Board next
> Wednesday.
> 
> I wanted to share this, as these might not yet be the right proposals,
> however I am putting them out there for feedback to start the discussion.
> 
> Thanks,
> 
> Ray K
> 
> 
> Experimental API
> 1.	APIs designated as experimental are not considered part of the ABI
> and may change without warning at any time.
> 2.	APIs designated as experimental must be marked depreciated for a
> least one quarterly release before removal.
> 3.	APIs designated as experimental will no longer automatically graduate
> to core after one release, they may stay experimental until their author
> and the maintainer agree that graduation is appropriate.
> 
> Core API (non-experimental API)
> 4.	APIs designated as core must be depreciated for a least two years
> before removal, to facilitate the continued compatibility with LTS
> releases. A final removal notice will be published to the DPDK Mailing
> List, and if there are no strong objections only then an API may be
> removed.
> 5.	APIs designated as core may be changed as follows:-
> 5.a	The change proposer must demonstrated that the change has a
> supporting use case and could not be achieved in any other way.
> 5.b	ABI version compatibility must be retained, as described below.

Hi Ray,

My somewhat rambly 2 cents :)

While i think some solution has to be found for the situation, we also 
have to balance this against speed of development and new features rollout.

For example, let's consider what i am intimately familiar with - the 
memory rework. I have made enormous efforts to ensure that pre-18.05 and 
post-18.05 remain as ABI/API compatible as possible, but there were a 
couple of API calls that were removed, and there couldn't have been any 
replacements (these API's were exposing internal structures that 
shouldn't have been exposed in the first place), and 18.05 also broke 
the ABI compatibility, because there was no way to do it without it 
(shared internal structures needed to change in part to support 
multiprocess).

So, if i understand your proposal correctly, assuming a 2-year waiting 
period for the deprecation of core API's, you would essentially still be 
waiting for the memory rework to land for a year more. Moreover, even 
*after* it has landed, there was a continuous stream of improvements and 
bugfixes, some of which has broke ABI compatibility as well. Some of 
them were my fault (as in, i could've foreseen the need for those 
changes, but didn't), but others came as a result of people using these 
new features in the wild and reporting issues/problems/suggestions - i 
am but one man, after all. Plus, you know, there's only 24 hours in a 
day, and some stuff takes time to implement :)

Since this rework goes right at the heart of DPDK (arguably there isn't 
a more "core" API than memory!), there is no (sane) way in the universe 
to 1) keep backwards compatibility for this, or 2) keep two parallel 
versions of it. We also need to test all that, and, to be honest, one 
validation cycle for a release wouldn't be enough to figure out all of 
the kinks and implications of such a case. It was really great that 
memory rework has landed in 18.05 and we had time to improve and prepare 
it for an 18.11 LTS - i think everyone can say that it's in much better 
shape in 18.11 than it was in 18.05, but if we couldn't do an ABI break 
here or there, this rate of improvements would have slowed down 
significantly.

Now, i understand that this is probably a highly exceptional case, but 
i'm sure that maintainers of other parts of DPDK will have their own 
examples of similar things happening.

I have no idea what a proper solution would look like. Any "splitting" 
of the trees into "experimental" vs. "stable" will end up causing the 
same issue - people choose to use stable over experimental because, 
well, it's more stable, and new/experimental features don't get tested 
as much because no one runs the thing in the first place.

TL;DR we have to be careful not to constrain the pace of 
development/bugfixing just for the sake of having a stable API/ABI :)

> 
> Shared Libraries
> 6.	DPDK will move to shared libraries & dynamic linking by default, to
> accommodate greater use of ABI versioning by DPDK consumers.
> 
> ABI Versioning
> 7.	New quarterly releases of DPDK will remain ABI compatible with the
> most recent DPDK LTS release.
> (e.g. DPDK 19.08 will remain ABI compatible with DPDK LTS 18.11).
> 8.	New DPDK LTS releases will remain ABI compatible with the previous
> two DPDK LTS releases.
> (e.g. DPDK 20.11 will be ABI compatible with DPDK 19.11 and DPDK 18.11,
> DPDK 21.11 will be ABI compatible with DPDK 20.11 and DPDK 19.11 etc)
> 8. & 9. will be achieved with ABI symbol versioning.
> 
> 


-- 
Thanks,
Anatoly

Re: DPDK ABI/API Stability

[Kevin Traynor]

On 03/04/2019 16:42, Ray Kinsella wrote:
> Hi folks,
> 
> Recently I started a discussion with the DPDK Technical Board on DPDK
> ABI/API stability. This was born out informal feedback I had received
> from a number of users of DPDK about ABI churn. In turn this feedback
> then prompted an ABI analysis of DPDK using tools from abi-laboratory.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> 
> I guess the short story is that DPDK ABI hasn't really settled down as
> the project has matured. If you take a look at the “Backward Compat.”
> column which measures ABI compatibility compared to the previous
> releases, you will see significant churn in the ABI over successive
> releases since v16.04.
> 
> Now compare DPDK to GStreamer as an example of a very mature project
> with a similar intent, a framework for building applications, and which
> enjoys a very stable API.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> 
> The DPDK ABI churn has the following affects for users:-
> 
> 1. The churn obliges users of DPDK to commit to a constant
> re-integration and re-validation effort for new versions of DPDK. This
> effort from their perspective may not add value to their consuming
> project, particular if they are only updating to "stay current".
> 2. The churn encourages users of DPDK to slip versions, putting off
> reintegration to later, building up technical debt and causing their
> projects to miss support for new hardware or features.
> 3. It makes DPDK different to almost every other system library and
> framework that an operating systems might ship. This makes DPDK trickier
> to dynamically link against, package and maintain for OS maintainers.
> 

+1

> In order to address this issue, I have put together the minimal set of
> concrete proposals below for discussion at the Technical Board next
> Wednesday.
> 
> I wanted to share this, as these might not yet be the right proposals,
> however I am putting them out there for feedback to start the discussion.
> 
> Thanks,
> 
> Ray K
> 
> 
> Experimental API
> 1.	APIs designated as experimental are not considered part of the ABI
> and may change without warning at any time.
> 2.	APIs designated as experimental must be marked depreciated for a
> least one quarterly release before removal.
> 3.	APIs designated as experimental will no longer automatically graduate
> to core after one release, they may stay experimental until their author
> and the maintainer agree that graduation is appropriate.
> 

I don't think they automatically graduate in the current scheme, so it
would seem to be no change to the process.

One thing to note is that in the current experimental API scheme those
API are tagged as deprecated which can cause problems with application
build/CI if they are used. But this is more related to implementation
rather than the classification indicating the API may change. It might
need to change if there was a trend for keeping API as experimental for
a longer time.

> Core API (non-experimental API)
> 4.	APIs designated as core must be depreciated for a least two years
> before removal, to facilitate the continued compatibility with LTS
> releases. A final removal notice will be published to the DPDK Mailing
> List, and if there are no strong objections only then an API may be
> removed.
> 5.	APIs designated as core may be changed as follows:-

> 5.a	The change proposer must demonstrated that the change has a
> supporting use case and could not be achieved in any other way.

I agree with the sentiment but it needs real buy in from authors.
Otherwise the burden falls on the reviewer to spend time
exploring/finding a way not to change the API.

> 5.b	ABI version compatibility must be retained, as described below.
> 
> Shared Libraries
> 6.	DPDK will move to shared libraries & dynamic linking by default, to
> accommodate greater use of ABI versioning by DPDK consumers.
> 
> ABI Versioning
> 7.	New quarterly releases of DPDK will remain ABI compatible with the
> most recent DPDK LTS release.
> (e.g. DPDK 19.08 will remain ABI compatible with DPDK LTS 18.11).
> 8.	New DPDK LTS releases will remain ABI compatible with the previous
> two DPDK LTS releases.
> (e.g. DPDK 20.11 will be ABI compatible with DPDK 19.11 and DPDK 18.11,
> DPDK 21.11 will be ABI compatible with DPDK 20.11 and DPDK 19.11 etc)
> 8. & 9. will be achieved with ABI symbol versioning.
> 

Re: [dpdk-techboard] DPDK ABI/API Stability

[Bruce Richardson]

On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> > Hi folks,
> > 
> > Recently I started a discussion with the DPDK Technical Board on DPDK
> > ABI/API stability. This was born out informal feedback I had received
> > from a number of users of DPDK about ABI churn. In turn this feedback
> > then prompted an ABI analysis of DPDK using tools from abi-laboratory.
> > 
> > https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> > 
> > I guess the short story is that DPDK ABI hasn't really settled down as
> > the project has matured. If you take a look at the “Backward Compat.”
> > column which measures ABI compatibility compared to the previous
> > releases, you will see significant churn in the ABI over successive
> > releases since v16.04.
> > 
> > Now compare DPDK to GStreamer as an example of a very mature project
> > with a similar intent, a framework for building applications, and which
> > enjoys a very stable API.
> > 
> > https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> > 
> > The DPDK ABI churn has the following affects for users:-
> > 
> > 1. The churn obliges users of DPDK to commit to a constant
> > re-integration and re-validation effort for new versions of DPDK. This
> > effort from their perspective may not add value to their consuming
> > project, particular if they are only updating to "stay current".
> > 2. The churn encourages users of DPDK to slip versions, putting off
> > reintegration to later, building up technical debt and causing their
> > projects to miss support for new hardware or features.
> > 3. It makes DPDK different to almost every other system library and
> > framework that an operating systems might ship. This makes DPDK trickier
> > to dynamically link against, package and maintain for OS maintainers.
> > 
> > In order to address this issue, I have put together the minimal set of
> > concrete proposals below for discussion at the Technical Board next
> > Wednesday.
> > 
> > I wanted to share this, as these might not yet be the right proposals,
> > however I am putting them out there for feedback to start the discussion.
> > 
> > Thanks,
> > 
> > Ray K
> > 
> > 
> > Experimental API
> > 1.	APIs designated as experimental are not considered part of the ABI
> > and may change without warning at any time.
> > 2.	APIs designated as experimental must be marked depreciated for a
> > least one quarterly release before removal.
> > 3.	APIs designated as experimental will no longer automatically graduate
> > to core after one release, they may stay experimental until their author
> > and the maintainer agree that graduation is appropriate.
> > 
> > Core API (non-experimental API)
> > 4.	APIs designated as core must be depreciated for a least two years
> > before removal, to facilitate the continued compatibility with LTS
> > releases. A final removal notice will be published to the DPDK Mailing
> > List, and if there are no strong objections only then an API may be
> > removed.
> > 5.	APIs designated as core may be changed as follows:-
> > 5.a	The change proposer must demonstrated that the change has a
> > supporting use case and could not be achieved in any other way.
> > 5.b	ABI version compatibility must be retained, as described below.
> 
> Hi Ray,
> 
> My somewhat rambly 2 cents :)
> 
> While i think some solution has to be found for the situation, we also have
> to balance this against speed of development and new features rollout.
> 
> For example, let's consider what i am intimately familiar with - the memory
> rework. I have made enormous efforts to ensure that pre-18.05 and post-18.05
> remain as ABI/API compatible as possible, but there were a couple of API
> calls that were removed, and there couldn't have been any replacements
> (these API's were exposing internal structures that shouldn't have been
> exposed in the first place), and 18.05 also broke the ABI compatibility,
> because there was no way to do it without it (shared internal structures
> needed to change in part to support multiprocess).
> 
> So, if i understand your proposal correctly, assuming a 2-year waiting
> period for the deprecation of core API's, you would essentially still be
> waiting for the memory rework to land for a year more. Moreover, even
> *after* it has landed, there was a continuous stream of improvements and
> bugfixes, some of which has broke ABI compatibility as well. Some of them
> were my fault (as in, i could've foreseen the need for those changes, but
> didn't), but others came as a result of people using these new features in
> the wild and reporting issues/problems/suggestions - i am but one man, after
> all. Plus, you know, there's only 24 hours in a day, and some stuff takes
> time to implement :)
> 
> Since this rework goes right at the heart of DPDK (arguably there isn't a
> more "core" API than memory!), there is no (sane) way in the universe to 1)
> keep backwards compatibility for this, or 2) keep two parallel versions of
> it. We also need to test all that, and, to be honest, one validation cycle
> for a release wouldn't be enough to figure out all of the kinks and
> implications of such a case. It was really great that memory rework has
> landed in 18.05 and we had time to improve and prepare it for an 18.11 LTS -
> i think everyone can say that it's in much better shape in 18.11 than it was
> in 18.05, but if we couldn't do an ABI break here or there, this rate of
> improvements would have slowed down significantly.
> 
> Now, i understand that this is probably a highly exceptional case, but i'm
> sure that maintainers of other parts of DPDK will have their own examples of
> similar things happening.
> 
> I have no idea what a proper solution would look like. Any "splitting" of
> the trees into "experimental" vs. "stable" will end up causing the same
> issue - people choose to use stable over experimental because, well, it's
> more stable, and new/experimental features don't get tested as much because
> no one runs the thing in the first place.
> 
> TL;DR we have to be careful not to constrain the pace of
> development/bugfixing just for the sake of having a stable API/ABI :)
> 

Actually, I think we *do* need to constrain the pace of development for the
sake of ABI stability. At this stage DPDK has been around for quite a
number of years and so should be considered a fairly mature project - it
should just start acting like it.

Now, in terms of features like the memory rework, that is indeed a case
that there was no alternative other than a massive ABI break. However, for
that rework there was a strong need for improvement in that area that we
can make the case for an ABI break to support it - and it is of a scale
that nothing other than an ABI change would do. For other areas and
examples, I doubt there are many in the last couple of years that are of
that scale.

My thoughts on the matter are:
1. I think we really need to do work to start hiding more of our data
structures - like what Stephen's latest RFC does. This hiding should reduce
the scope for ABI breaks.
2. Once done, I think we should commit to having an ABI break only in the
rarest of circumstances, and only with very large justification. I want us
to get to the point where DPDK releases can immediately be picked up by all
linux distros and rolled out because they are ABI compatible.

I'm not sure I like the idea of planned ABI break releases - that strikes
me as a plan where we end up with the same number of ABI breaks as before,
just balled into one release.

Question for Kevin, Luca and others who look at distro-packaging: is it the
case that each distro will only ship one version of DPDK, or is it possible
that if we have ABI breaks, a distro will provide two copies of DPDK
simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?


So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
ABI breaks, and having ABI breaks as a major event reserved only for
massive rework changes, such as major mbuf changes, or new memory layout or
similar.

Regards,
/Bruce

Re: [dpdk-techboard] DPDK ABI/API Stability

[Luca Boccassi]

On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> > On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> > > Hi folks,
> > > 
> > > Recently I started a discussion with the DPDK Technical Board on
> > > DPDK
> > > ABI/API stability. This was born out informal feedback I had
> > > received
> > > from a number of users of DPDK about ABI churn. In turn this
> > > feedback
> > > then prompted an ABI analysis of DPDK using tools from abi-
> > > laboratory.
> > > 
> > > https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> > > 
> > > 
> > > I guess the short story is that DPDK ABI hasn't really settled
> > > down as
> > > the project has matured. If you take a look at the “Backward
> > > Compat.”
> > > column which measures ABI compatibility compared to the previous
> > > releases, you will see significant churn in the ABI over
> > > successive
> > > releases since v16.04.
> > > 
> > > Now compare DPDK to GStreamer as an example of a very mature
> > > project
> > > with a similar intent, a framework for building applications, and
> > > which
> > > enjoys a very stable API.
> > > 
> > > https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> > > 
> > > 
> > > The DPDK ABI churn has the following affects for users:-
> > > 
> > > 1. The churn obliges users of DPDK to commit to a constant
> > > re-integration and re-validation effort for new versions of DPDK.
> > > This
> > > effort from their perspective may not add value to their
> > > consuming
> > > project, particular if they are only updating to "stay current".
> > > 2. The churn encourages users of DPDK to slip versions, putting
> > > off
> > > reintegration to later, building up technical debt and causing
> > > their
> > > projects to miss support for new hardware or features.
> > > 3. It makes DPDK different to almost every other system library
> > > and
> > > framework that an operating systems might ship. This makes DPDK
> > > trickier
> > > to dynamically link against, package and maintain for OS
> > > maintainers.
> > > 
> > > In order to address this issue, I have put together the minimal
> > > set of
> > > concrete proposals below for discussion at the Technical Board
> > > next
> > > Wednesday.
> > > 
> > > I wanted to share this, as these might not yet be the right
> > > proposals,
> > > however I am putting them out there for feedback to start the
> > > discussion.
> > > 
> > > Thanks,
> > > 
> > > Ray K
> > > 
> > > 
> > > Experimental API
> > > 1.	APIs designated as experimental are not considered part
> > > of the ABI
> > > and may change without warning at any time.
> > > 2.	APIs designated as experimental must be marked
> > > depreciated for a
> > > least one quarterly release before removal.
> > > 3.	APIs designated as experimental will no longer
> > > automatically graduate
> > > to core after one release, they may stay experimental until their
> > > author
> > > and the maintainer agree that graduation is appropriate.
> > > 
> > > Core API (non-experimental API)
> > > 4.	APIs designated as core must be depreciated for a least
> > > two years
> > > before removal, to facilitate the continued compatibility with
> > > LTS
> > > releases. A final removal notice will be published to the DPDK
> > > Mailing
> > > List, and if there are no strong objections only then an API may
> > > be
> > > removed.
> > > 5.	APIs designated as core may be changed as follows:-
> > > 5.a	The change proposer must demonstrated that the change
> > > has a
> > > supporting use case and could not be achieved in any other way.
> > > 5.b	ABI version compatibility must be retained, as
> > > described below.
> > 
> > Hi Ray,
> > 
> > My somewhat rambly 2 cents :)
> > 
> > While i think some solution has to be found for the situation, we
> > also have
> > to balance this against speed of development and new features
> > rollout.
> > 
> > For example, let's consider what i am intimately familiar with -
> > the memory
> > rework. I have made enormous efforts to ensure that pre-18.05 and
> > post-18.05
> > remain as ABI/API compatible as possible, but there were a couple
> > of API
> > calls that were removed, and there couldn't have been any
> > replacements
> > (these API's were exposing internal structures that shouldn't have
> > been
> > exposed in the first place), and 18.05 also broke the ABI
> > compatibility,
> > because there was no way to do it without it (shared internal
> > structures
> > needed to change in part to support multiprocess).
> > 
> > So, if i understand your proposal correctly, assuming a 2-year
> > waiting
> > period for the deprecation of core API's, you would essentially
> > still be
> > waiting for the memory rework to land for a year more. Moreover,
> > even
> > *after* it has landed, there was a continuous stream of
> > improvements and
> > bugfixes, some of which has broke ABI compatibility as well. Some
> > of them
> > were my fault (as in, i could've foreseen the need for those
> > changes, but
> > didn't), but others came as a result of people using these new
> > features in
> > the wild and reporting issues/problems/suggestions - i am but one
> > man, after
> > all. Plus, you know, there's only 24 hours in a day, and some stuff
> > takes
> > time to implement :)
> > 
> > Since this rework goes right at the heart of DPDK (arguably there
> > isn't a
> > more "core" API than memory!), there is no (sane) way in the
> > universe to 1)
> > keep backwards compatibility for this, or 2) keep two parallel
> > versions of
> > it. We also need to test all that, and, to be honest, one
> > validation cycle
> > for a release wouldn't be enough to figure out all of the kinks and
> > implications of such a case. It was really great that memory rework
> > has
> > landed in 18.05 and we had time to improve and prepare it for an
> > 18.11 LTS -
> > i think everyone can say that it's in much better shape in 18.11
> > than it was
> > in 18.05, but if we couldn't do an ABI break here or there, this
> > rate of
> > improvements would have slowed down significantly.
> > 
> > Now, i understand that this is probably a highly exceptional case,
> > but i'm
> > sure that maintainers of other parts of DPDK will have their own
> > examples of
> > similar things happening.
> > 
> > I have no idea what a proper solution would look like. Any
> > "splitting" of
> > the trees into "experimental" vs. "stable" will end up causing the
> > same
> > issue - people choose to use stable over experimental because,
> > well, it's
> > more stable, and new/experimental features don't get tested as much
> > because
> > no one runs the thing in the first place.
> > 
> > TL;DR we have to be careful not to constrain the pace of
> > development/bugfixing just for the sake of having a stable API/ABI
> > :)
> > 
> 
> Actually, I think we *do* need to constrain the pace of development
> for the
> sake of ABI stability. At this stage DPDK has been around for quite a
> number of years and so should be considered a fairly mature project -
> it
> should just start acting like it.
> 
> Now, in terms of features like the memory rework, that is indeed a
> case
> that there was no alternative other than a massive ABI break.
> However, for
> that rework there was a strong need for improvement in that area that
> we
> can make the case for an ABI break to support it - and it is of a
> scale
> that nothing other than an ABI change would do. For other areas and
> examples, I doubt there are many in the last couple of years that are
> of
> that scale.

Fully agree.

It's normal for new project, big and small, to start without a
stability promise as development ramps up, and then steer toward
stability as the user base grows. Sometimes the switch is made explicit
by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
I don't think we crossed that boundary yet in this project, and I
believe that's the main question Ray was trying to raise: has the time
finally come for DPDK to do this phase shift?

Of course it comes with a price for all developers, and that's again
common.

> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should
> reduce
> the scope for ABI breaks.

Yes, I'm a big fan of accessors and opaque structs.

> 2. Once done, I think we should commit to having an ABI break only in
> the
> rarest of circumstances, and only with very large justification. I
> want us
> to get to the point where DPDK releases can immediately be picked up
> by all
> linux distros and rolled out because they are ABI compatible.
> 
> I'm not sure I like the idea of planned ABI break releases - that
> strikes
> me as a plan where we end up with the same number of ABI breaks as
> before,
> just balled into one release.

I think that was intended as a compromise, especially as we move from
one model to the other, and more of a "if a breakage has to happen, it
must be in the X release" rather than "let's always break in the X
release" :-)

> Question for Kevin, Luca and others who look at distro-packaging: is
> it the
> case that each distro will only ship one version of DPDK, or is it
> possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?

We can ship multiple versions, although it's more work so there should
be a good reason to do it. At the moment in Debian and Ubuntu we don't,
and we tend to ship whatever the latest LTS version is at the distro
freeze milestone - for example Debian 10 which will be released soon
(TM) will have 18.11.0.

> So, in short, I'm generally in favour of a zero-tolerance approach
> for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory
> layout or
> similar.
> 
> Regards,
> /Bruce
> 
-- 
Kind regards,
Luca Boccassi

Re: [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 04/04/2019 11:54, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>> Hi folks,
>>>
[SNIP]
>>
>> Hi Ray,
>>
>> My somewhat rambly 2 cents :)
>>
>> While i think some solution has to be found for the situation, we also have
>> to balance this against speed of development and new features rollout.
>>
>> For example, let's consider what i am intimately familiar with - the memory
>> rework. I have made enormous efforts to ensure that pre-18.05 and post-18.05
>> remain as ABI/API compatible as possible, but there were a couple of API
>> calls that were removed, and there couldn't have been any replacements
>> (these API's were exposing internal structures that shouldn't have been
>> exposed in the first place), and 18.05 also broke the ABI compatibility,
>> because there was no way to do it without it (shared internal structures
>> needed to change in part to support multiprocess).
>>
>> So, if i understand your proposal correctly, assuming a 2-year waiting
>> period for the deprecation of core API's, you would essentially still be
>> waiting for the memory rework to land for a year more. Moreover, even
>> *after* it has landed, there was a continuous stream of improvements and
>> bugfixes, some of which has broke ABI compatibility as well. Some of them
>> were my fault (as in, i could've foreseen the need for those changes, but
>> didn't), but others came as a result of people using these new features in
>> the wild and reporting issues/problems/suggestions - i am but one man, after
>> all. Plus, you know, there's only 24 hours in a day, and some stuff takes
>> time to implement :)
>>
>> Since this rework goes right at the heart of DPDK (arguably there isn't a
>> more "core" API than memory!), there is no (sane) way in the universe to 1)
>> keep backwards compatibility for this, or 2) keep two parallel versions of
>> it. We also need to test all that, and, to be honest, one validation cycle
>> for a release wouldn't be enough to figure out all of the kinks and
>> implications of such a case. It was really great that memory rework has
>> landed in 18.05 and we had time to improve and prepare it for an 18.11 LTS -
>> i think everyone can say that it's in much better shape in 18.11 than it was
>> in 18.05, but if we couldn't do an ABI break here or there, this rate of
>> improvements would have slowed down significantly.
>>
>> Now, i understand that this is probably a highly exceptional case, but i'm
>> sure that maintainers of other parts of DPDK will have their own examples of
>> similar things happening.
>>
>> I have no idea what a proper solution would look like. Any "splitting" of
>> the trees into "experimental" vs. "stable" will end up causing the same
>> issue - people choose to use stable over experimental because, well, it's
>> more stable, and new/experimental features don't get tested as much because
>> no one runs the thing in the first place.
>>
>> TL;DR we have to be careful not to constrain the pace of
>> development/bugfixing just for the sake of having a stable API/ABI :)
>>
> 
> Actually, I think we *do* need to constrain the pace of development for the
> sake of ABI stability. At this stage DPDK has been around for quite a
> number of years and so should be considered a fairly mature project - it
> should just start acting like it.

I 100% agree.

If you break your users stuff regularly enough, they will eventually
start looking around for an alternative that doesn't break their stuff
quiet so regularly.

We often use the pace of innovation in DPDK as justification for ABI/API
breakages, but that approach is a real rarity among the Open Source
community. I can't think of any mature project off-hand that share's it.

I would ask is Linux any less innovative because they offer a stable API
and have an absolute commitment to never breaking userspace? Would Linux
have ever been as popular as it is today it they broke userspace every
quarter?

They reality is that they (Linux) find workarounds and compromise
because there is an uber-maintainer Linus who had a strong ethos from
the start not to break their users stuff - we need the same ethos in DPDK.

> 
> Now, in terms of features like the memory rework, that is indeed a case
> that there was no alternative other than a massive ABI break. However, for
> that rework there was a strong need for improvement in that area that we
> can make the case for an ABI break to support it - and it is of a scale
> that nothing other than an ABI change would do. For other areas and
> examples, I doubt there are many in the last couple of years that are of
> that scale.

I would also be inclined to agree with Bruce's points on memory rework
was somewhat of an outlier, we don't see many like it.
> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should reduce
> the scope for ABI breaks.
> 2. Once done, I think we should commit to having an ABI break only in the
> rarest of circumstances, and only with very large justification. I want us
> to get to the point where DPDK releases can immediately be picked up by all
> linux distros and rolled out because they are ABI compatible.

The work that Anatoly describes removing APIs that exposed internal
structures and Stephen H's RFC similarly are good examples of the kind
of work required to prepare for this change. We need to take a good look
at the API and reduce the number of unnecessary internal structures
exposed.

I never expected it going to to be a big bang - but is a definite
direction we need to move towards over the next few release.

> I'm not sure I like the idea of planned ABI break releases - that strikes
> me as a plan where we end up with the same number of ABI breaks as before,
> just balled into one release.
> 

I agree, with this approach you just delaying the evil day and when that
day comes it will 10-times more painful. This is no substitute for a
don't break our users stuff ethos.

> Question for Kevin, Luca and others who look at distro-packaging: is it the
> case that each distro will only ship one version of DPDK, or is it possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
> 
> 
> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory layout or
> similar.
> 
> Regards,
> /Bruce
> 

Re: [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 04/04/2019 13:02, Luca Boccassi wrote:
> On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
[SNIP]
>>>
>>
>> Actually, I think we *do* need to constrain the pace of development
>> for the
>> sake of ABI stability. At this stage DPDK has been around for quite a
>> number of years and so should be considered a fairly mature project -
>> it
>> should just start acting like it.
>>
>> Now, in terms of features like the memory rework, that is indeed a
>> case
>> that there was no alternative other than a massive ABI break.
>> However, for
>> that rework there was a strong need for improvement in that area that
>> we
>> can make the case for an ABI break to support it - and it is of a
>> scale
>> that nothing other than an ABI change would do. For other areas and
>> examples, I doubt there are many in the last couple of years that are
>> of
>> that scale.
> 
> Fully agree.
> 
> It's normal for new project, big and small, to start without a
> stability promise as development ramps up, and then steer toward
> stability as the user base grows. Sometimes the switch is made explicit
> by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
> I don't think we crossed that boundary yet in this project, and I
> believe that's the main question Ray was trying to raise: has the time
> finally come for DPDK to do this phase shift?

Yes - we never had a 1.0, where we cut an API and stood behind it
similar to GStreamer. There a number of reasons why this happened not
worth going into, however you make the point very well Luca - this phase
shift is long over due.

> 
> Of course it comes with a price for all developers, and that's again
> common.

Agreed - nothing is for free.
The question is this something we value and is it something worth doing.

> 
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should
>> reduce
>> the scope for ABI breaks.
> 
> Yes, I'm a big fan of accessors and opaque structs.

+1, me too.

> 
>> 2. Once done, I think we should commit to having an ABI break only in
>> the
>> rarest of circumstances, and only with very large justification. I
>> want us
>> to get to the point where DPDK releases can immediately be picked up
>> by all
>> linux distros and rolled out because they are ABI compatible.
>>
>> I'm not sure I like the idea of planned ABI break releases - that
>> strikes
>> me as a plan where we end up with the same number of ABI breaks as
>> before,
>> just balled into one release.
> 
> I think that was intended as a compromise, especially as we move from
> one model to the other, and more of a "if a breakage has to happen, it
> must be in the X release" rather than "let's always break in the X
> release" :-)
> 
>> Question for Kevin, Luca and others who look at distro-packaging: is
>> it the
>> case that each distro will only ship one version of DPDK, or is it
>> possible
>> that if we have ABI breaks, a distro will provide two copies of DPDK
>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
> 
> We can ship multiple versions, although it's more work so there should
> be a good reason to do it. 

Well you already kind of do right.
You ship 16.11.8 with Debian 9 and then 18.11 with Debian 9 backports.

> At the moment in Debian and Ubuntu we don't,
> and we tend to ship whatever the latest LTS version is at the distro
> freeze milestone - for example Debian 10 which will be released soon
> (TM) will have 18.11.0.

Presumably when 19.11 arrives, it will land in Debian 10 backports
similarly.

I assume anything that lands in backports is not guaranteed to be ABI
compatible with stable?

> 
>> So, in short, I'm generally in favour of a zero-tolerance approach
>> for DPDK
>> ABI breaks, and having ABI breaks as a major event reserved only for
>> massive rework changes, such as major mbuf changes, or new memory
>> layout or
>> similar.
>>
>> Regards,
>> /Bruce
>>

Re: [dpdk-techboard] DPDK ABI/API Stability

[Bruce Richardson]

On Thu, Apr 04, 2019 at 02:05:27PM +0100, Ray Kinsella wrote:
> 
> 
> On 04/04/2019 13:02, Luca Boccassi wrote:
> > On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> >> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> [SNIP]
> >>>
> >>
> >> Actually, I think we *do* need to constrain the pace of development
> >> for the
> >> sake of ABI stability. At this stage DPDK has been around for quite a
> >> number of years and so should be considered a fairly mature project -
> >> it
> >> should just start acting like it.
> >>
> >> Now, in terms of features like the memory rework, that is indeed a
> >> case
> >> that there was no alternative other than a massive ABI break.
> >> However, for
> >> that rework there was a strong need for improvement in that area that
> >> we
> >> can make the case for an ABI break to support it - and it is of a
> >> scale
> >> that nothing other than an ABI change would do. For other areas and
> >> examples, I doubt there are many in the last couple of years that are
> >> of
> >> that scale.
> > 
> > Fully agree.
> > 
> > It's normal for new project, big and small, to start without a
> > stability promise as development ramps up, and then steer toward
> > stability as the user base grows. Sometimes the switch is made explicit
> > by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
> > I don't think we crossed that boundary yet in this project, and I
> > believe that's the main question Ray was trying to raise: has the time
> > finally come for DPDK to do this phase shift?
> 
> Yes - we never had a 1.0, where we cut an API and stood behind it
> similar to GStreamer. There a number of reasons why this happened not
> worth going into, however you make the point very well Luca - this phase
> shift is long over due.
> 
> > 
> > Of course it comes with a price for all developers, and that's again
> > common.
> 
> Agreed - nothing is for free.
> The question is this something we value and is it something worth doing.
> 
> > 
> >> My thoughts on the matter are:
> >> 1. I think we really need to do work to start hiding more of our data
> >> structures - like what Stephen's latest RFC does. This hiding should
> >> reduce
> >> the scope for ABI breaks.
> > 
> > Yes, I'm a big fan of accessors and opaque structs.
> 
> +1, me too.
> 
I would be too, with certain exceptions - rte_mbuf for one. Any structures
that are used by applications cannot be made opaque, as apps do not want to
pay the cost of having to call a function every time they want to access
something from one of those structures.

Thankfully for us, we have plenty of other structures that we can work on
in the meantime that can be made private to avoid ABI breaks! :-) I suggest
we work through those first, allowing us to hone our ABI-break avoidance
skills.

/Bruce

Re: DPDK ABI/API Stability

[Ray Kinsella]

On 04/04/2019 10:47, Kevin Traynor wrote:
> On 03/04/2019 16:42, Ray Kinsella wrote:
>> Hi folks,
>>
[SNIP]
>>
>> The DPDK ABI churn has the following affects for users:-
>>
>> 1. The churn obliges users of DPDK to commit to a constant
>> re-integration and re-validation effort for new versions of DPDK. This
>> effort from their perspective may not add value to their consuming
>> project, particular if they are only updating to "stay current".
>> 2. The churn encourages users of DPDK to slip versions, putting off
>> reintegration to later, building up technical debt and causing their
>> projects to miss support for new hardware or features.
>> 3. It makes DPDK different to almost every other system library and
>> framework that an operating systems might ship. This makes DPDK trickier
>> to dynamically link against, package and maintain for OS maintainers.
>>
> 
> +1
> 
>> In order to address this issue, I have put together the minimal set of
>> concrete proposals below for discussion at the Technical Board next
>> Wednesday.
>>
>> I wanted to share this, as these might not yet be the right proposals,
>> however I am putting them out there for feedback to start the discussion.
>>
>> Thanks,
>>
>> Ray K
>>
>>
>> Experimental API
>> 1.	APIs designated as experimental are not considered part of the ABI
>> and may change without warning at any time.
>> 2.	APIs designated as experimental must be marked depreciated for a
>> least one quarterly release before removal.
>> 3.	APIs designated as experimental will no longer automatically graduate
>> to core after one release, they may stay experimental until their author
>> and the maintainer agree that graduation is appropriate.
>>
> 
> I don't think they automatically graduate in the current scheme, so it
> would seem to be no change to the process.

You are correct ...

"New APIs will be marked as experimental for at least one release to
allow any issues found by users of the new API to be fixed quickly."
https://doc.dpdk.org/guides-18.11/contributing/versioning.html


> 
> One thing to note is that in the current experimental API scheme those
> API are tagged as deprecated which can cause problems with application
> build/CI if they are used. But this is more related to implementation
> rather than the classification indicating the API may change. It might
> need to change if there was a trend for keeping API as experimental for
> a longer time.

Others have said this to me also - we need to make experimental less
scary - so people won't be quiet as eager to remove the tag. Depreciated
should stay scary however.

> 
>> Core API (non-experimental API)
>> 4.	APIs designated as core must be depreciated for a least two years
>> before removal, to facilitate the continued compatibility with LTS
>> releases. A final removal notice will be published to the DPDK Mailing
>> List, and if there are no strong objections only then an API may be
>> removed.
>> 5.	APIs designated as core may be changed as follows:-
> 
>> 5.a	The change proposer must demonstrated that the change has a
>> supporting use case and could not be achieved in any other way.
> 
> I agree with the sentiment but it needs real buy in from authors.

Well I would say it needs real buy in from the Maintainers, at the end
of the day they are the gatekeepers in this instance.

> Otherwise the burden falls on the reviewer to spend time
> exploring/finding a way not to change the API.

I think most of the time, Maintainers will have a good sense of when a
ABI/API change is actually warranted.

> 
>> 5.b	ABI version compatibility must be retained, as described below.
>>
>> Shared Libraries
>> 6.	DPDK will move to shared libraries & dynamic linking by default, to
>> accommodate greater use of ABI versioning by DPDK consumers.
>>
>> ABI Versioning
>> 7.	New quarterly releases of DPDK will remain ABI compatible with the
>> most recent DPDK LTS release.
>> (e.g. DPDK 19.08 will remain ABI compatible with DPDK LTS 18.11).
>> 8.	New DPDK LTS releases will remain ABI compatible with the previous
>> two DPDK LTS releases.
>> (e.g. DPDK 20.11 will be ABI compatible with DPDK 19.11 and DPDK 18.11,
>> DPDK 21.11 will be ABI compatible with DPDK 20.11 and DPDK 19.11 etc)
>> 8. & 9. will be achieved with ABI symbol versioning.
>>
> 

Re: [dpdk-techboard] DPDK ABI/API Stability

[Luca Boccassi]

On Thu, 2019-04-04 at 14:05 +0100, Ray Kinsella wrote:
> > > Question for Kevin, Luca and others who look at distro-packaging: 
> > > is
> > > it the
> > > case that each distro will only ship one version of DPDK, or is
> > > it
> > > possible
> > > that if we have ABI breaks, a distro will provide two copies of
> > > DPDK
> > > simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
> > 
> > We can ship multiple versions, although it's more work so there
> > should
> > be a good reason to do it. 
> 
> Well you already kind of do right.
> You ship 16.11.8 with Debian 9 and then 18.11 with Debian 9
> backports.

Yes indeed - backports has a different policy and is not part of
"Debian main" - for example, stability and security support are not
guaranteed. Also there is no guarantee of compatibility of co-
installability - while you can have the individual libraries from each
version at the same time, you can't have both libdpdk-dev packages
which are necessary to build an application. So you have to choose and
install either libdpdk-dev=16.11 or libdpdk-dev=18.11.

It is possible to have multiple versions in the same archive at the
same time, and with multiple -dev packages, and that's what I meant
when I mentioned it was more work.

> > At the moment in Debian and Ubuntu we don't,
> > and we tend to ship whatever the latest LTS version is at the
> > distro
> > freeze milestone - for example Debian 10 which will be released
> > soon
> > (TM) will have 18.11.0.
> 
> Presumably when 19.11 arrives, it will land in Debian 10 backports
> similarly.
> 
> I assume anything that lands in backports is not guaranteed to be ABI
> compatible with stable?

Yes, there's no guarantee.

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> 
> 
> On 04/04/2019 11:54, Bruce Richardson wrote:
>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>>> Hi folks,
>>>>
> [SNIP]
>>>
>>> Hi Ray,
>>>
>>> My somewhat rambly 2 cents :)
>>>
>>> While i think some solution has to be found for the situation, we also have
>>> to balance this against speed of development and new features rollout.
>>>
>>> For example, let's consider what i am intimately familiar with - the memory
>>> rework. I have made enormous efforts to ensure that pre-18.05 and post-18.05
>>> remain as ABI/API compatible as possible, but there were a couple of API
>>> calls that were removed, and there couldn't have been any replacements
>>> (these API's were exposing internal structures that shouldn't have been
>>> exposed in the first place), and 18.05 also broke the ABI compatibility,
>>> because there was no way to do it without it (shared internal structures
>>> needed to change in part to support multiprocess).
>>>
>>> So, if i understand your proposal correctly, assuming a 2-year waiting
>>> period for the deprecation of core API's, you would essentially still be
>>> waiting for the memory rework to land for a year more. Moreover, even
>>> *after* it has landed, there was a continuous stream of improvements and
>>> bugfixes, some of which has broke ABI compatibility as well. Some of them
>>> were my fault (as in, i could've foreseen the need for those changes, but
>>> didn't), but others came as a result of people using these new features in
>>> the wild and reporting issues/problems/suggestions - i am but one man, after
>>> all. Plus, you know, there's only 24 hours in a day, and some stuff takes
>>> time to implement :)
>>>
>>> Since this rework goes right at the heart of DPDK (arguably there isn't a
>>> more "core" API than memory!), there is no (sane) way in the universe to 1)
>>> keep backwards compatibility for this, or 2) keep two parallel versions of
>>> it. We also need to test all that, and, to be honest, one validation cycle
>>> for a release wouldn't be enough to figure out all of the kinks and
>>> implications of such a case. It was really great that memory rework has
>>> landed in 18.05 and we had time to improve and prepare it for an 18.11 LTS -
>>> i think everyone can say that it's in much better shape in 18.11 than it was
>>> in 18.05, but if we couldn't do an ABI break here or there, this rate of
>>> improvements would have slowed down significantly.
>>>
>>> Now, i understand that this is probably a highly exceptional case, but i'm
>>> sure that maintainers of other parts of DPDK will have their own examples of
>>> similar things happening.
>>>
>>> I have no idea what a proper solution would look like. Any "splitting" of
>>> the trees into "experimental" vs. "stable" will end up causing the same
>>> issue - people choose to use stable over experimental because, well, it's
>>> more stable, and new/experimental features don't get tested as much because
>>> no one runs the thing in the first place.
>>>
>>> TL;DR we have to be careful not to constrain the pace of
>>> development/bugfixing just for the sake of having a stable API/ABI :)
>>>
>>
>> Actually, I think we *do* need to constrain the pace of development for the
>> sake of ABI stability. At this stage DPDK has been around for quite a
>> number of years and so should be considered a fairly mature project - it
>> should just start acting like it.
> 
> I 100% agree.
> 
> If you break your users stuff regularly enough, they will eventually
> start looking around for an alternative that doesn't break their stuff
> quiet so regularly.
> 
> We often use the pace of innovation in DPDK as justification for ABI/API
> breakages, but that approach is a real rarity among the Open Source
> community. I can't think of any mature project off-hand that share's it.
> 
> I would ask is Linux any less innovative because they offer a stable API
> and have an absolute commitment to never breaking userspace? Would Linux
> have ever been as popular as it is today it they broke userspace every
> quarter?
> 
> They reality is that they (Linux) find workarounds and compromise
> because there is an uber-maintainer Linus who had a strong ethos from
> the start not to break their users stuff - we need the same ethos in DPDK.
> 
>>
>> Now, in terms of features like the memory rework, that is indeed a case
>> that there was no alternative other than a massive ABI break. However, for
>> that rework there was a strong need for improvement in that area that we
>> can make the case for an ABI break to support it - and it is of a scale
>> that nothing other than an ABI change would do. For other areas and
>> examples, I doubt there are many in the last couple of years that are of
>> that scale.
> 
> I would also be inclined to agree with Bruce's points on memory rework
> was somewhat of an outlier, we don't see many like it.
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should reduce
>> the scope for ABI breaks.
>> 2. Once done, I think we should commit to having an ABI break only in the
>> rarest of circumstances, and only with very large justification. I want us
>> to get to the point where DPDK releases can immediately be picked up by all
>> linux distros and rolled out because they are ABI compatible.
> 
> The work that Anatoly describes removing APIs that exposed internal
> structures and Stephen H's RFC similarly are good examples of the kind
> of work required to prepare for this change. We need to take a good look
> at the API and reduce the number of unnecessary internal structures
> exposed.
> 
> I never expected it going to to be a big bang - but is a definite
> direction we need to move towards over the next few release.

...in this case, we have to think long and hard about the fabled EAL 
rework/split, and in general *specifying* what is it that we want to 
support, and the use cases that we want to target. Right now there is a 
huge mountain of technical debt and kludges and workarounds that has 
accumulated over the years, and it exists precisely because "every 
change breaks someone's workflow".

For example, just in memory subsystem alone, we have legacy mem, because 
some use cases require huge amounts of contiguous memory, and not 
everyone is using VFIO; there's all of the 32-bit related workarounds 
and hacks; there's the single-file-segments stuff that could have been 
the default if not for the fact that we support kernels that don't 
support fallocate(); there are two different ways of doing in-memory 
mode, because not all kernels support memfd's; there is a gargantuan 
pile of workarounds (and "known issues", and just code in general) all 
over the DPDK codebase just to support our multiprocess model and all of 
the various warts that come with it.

In fact, i would even go as far as to say that *most* of EAL ABI breaks 
have been due to the fact that we store data in shared memory because of 
multiprocess - so there is simply no way we can change these internal 
data structures without ABI breaks, because even if they're not exposed 
through user-facing API, they are still exposed by virtue of secondary 
processes basically having an ABI contract with primary process instances.

So, if we are to cement our core API - we have to make a concrete effort 
to specify what goes and what stays, if we want it to be maintainable. 
The DPDK 1.0 specification, if you will :)

-- 
Thanks,
Anatoly

Re: [dpdk-techboard] DPDK ABI/API Stability

[Stephen Hemminger]

On Thu, 4 Apr 2019 11:54:47 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should reduce
> the scope for ABI breaks.
> 2. Once done, I think we should commit to having an ABI break only in the
> rarest of circumstances, and only with very large justification. I want us
> to get to the point where DPDK releases can immediately be picked up by all
> linux distros and rolled out because they are ABI compatible.

I would also like to propose "you get one ABI break" which means each
API/ABI change must hide more infrastructure than the last. This is
the "fool me once, ..." saying in API's.

For example,
the memory rework it would have been good if the structure of mempools etc
were hidden inside EAL and not exposed. but as usual hindsight is 20/20

Re: [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 04-Apr-19 4:51 PM, Stephen Hemminger wrote:
> On Thu, 4 Apr 2019 11:54:47 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should reduce
>> the scope for ABI breaks.
>> 2. Once done, I think we should commit to having an ABI break only in the
>> rarest of circumstances, and only with very large justification. I want us
>> to get to the point where DPDK releases can immediately be picked up by all
>> linux distros and rolled out because they are ABI compatible.
> 
> I would also like to propose "you get one ABI break" which means each
> API/ABI change must hide more infrastructure than the last. This is
> the "fool me once, ..." saying in API's.
> 
> For example,
> the memory rework it would have been good if the structure of mempools etc
> were hidden inside EAL and not exposed. but as usual hindsight is 20/20
> 

Mempools is not part of "memory rework" - it's a separate library built 
on top of EAL's memory subsystem :) When i talk about "memory API's", i 
mean memzone/malloc and friends, not mempool.

-- 
Thanks,
Anatoly

Re: [dpdk-techboard] DPDK ABI/API Stability

[Kevin Traynor]

On 04/04/2019 11:54, Bruce Richardson wrote:
<snip>

>
> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should reduce
> the scope for ABI breaks.
> 2. Once done, I think we should commit to having an ABI break only in the
> rarest of circumstances, and only with very large justification. I want us
> to get to the point where DPDK releases can immediately be picked up by all
> linux distros and rolled out because they are ABI compatible.
> 

Maybe techboard should explicitly approve ABI breaks and new APIs (or
APIs at transition from experimental to core). Just as a way to get more
eyeballs and scrutiny on them.

> I'm not sure I like the idea of planned ABI break releases - that strikes
> me as a plan where we end up with the same number of ABI breaks as before,
> just balled into one release.
> 
> Question for Kevin, Luca and others who look at distro-packaging: is it the
> case that each distro will only ship one version of DPDK, or is it possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
> 

It would probably double validation and maintenance, so it would require
a lot of extra effort.

> 
> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory layout or
> similar.
> 
> Regards,
> /Bruce
> 

Re: [dpdk-techboard] DPDK ABI/API Stability

[Wiles, Keith]

> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
> 
> On 04/04/2019 11:54, Bruce Richardson wrote:
> <snip>
> 
>> 
>> My thoughts on the matter are:
>> 1. I think we really need to do work to start hiding more of our data
>> structures - like what Stephen's latest RFC does. This hiding should reduce
>> the scope for ABI breaks.
>> 2. Once done, I think we should commit to having an ABI break only in the
>> rarest of circumstances, and only with very large justification. I want us
>> to get to the point where DPDK releases can immediately be picked up by all
>> linux distros and rolled out because they are ABI compatible.
>> 
> 
> Maybe techboard should explicitly approve ABI breaks and new APIs (or
> APIs at transition from experimental to core). Just as a way to get more
> eyeballs and scrutiny on them.

ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
> 
>> I'm not sure I like the idea of planned ABI break releases - that strikes
>> me as a plan where we end up with the same number of ABI breaks as before,
>> just balled into one release.
>> 
>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>> case that each distro will only ship one version of DPDK, or is it possible
>> that if we have ABI breaks, a distro will provide two copies of DPDK
>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>> 
> 
> It would probably double validation and maintenance, so it would require
> a lot of extra effort.
> 
>> 
>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>> ABI breaks, and having ABI breaks as a major event reserved only for
>> massive rework changes, such as major mbuf changes, or new memory layout or
>> similar.
>> 
>> Regards,
>> /Bruce
>> 
> 

Regards,
Keith

Re: [dpdk-techboard] DPDK ABI/API Stability

[Kevin Traynor]

On 04/04/2019 20:08, Wiles, Keith wrote:
> 
> 
>> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>>
>> On 04/04/2019 11:54, Bruce Richardson wrote:
>> <snip>
>>
>>>
>>> My thoughts on the matter are:
>>> 1. I think we really need to do work to start hiding more of our data
>>> structures - like what Stephen's latest RFC does. This hiding should reduce
>>> the scope for ABI breaks.
>>> 2. Once done, I think we should commit to having an ABI break only in the
>>> rarest of circumstances, and only with very large justification. I want us
>>> to get to the point where DPDK releases can immediately be picked up by all
>>> linux distros and rolled out because they are ABI compatible.
>>>
>>
>> Maybe techboard should explicitly approve ABI breaks and new APIs (or
>> APIs at transition from experimental to core). Just as a way to get more
>> eyeballs and scrutiny on them.
> 
> ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
>>

We'll only find out if they are bad when they need ABI breaks later :-)

My point is a good way to avoid future ABI breaks is to have more
reviews on the APIs in the first place. Techboard approval might be one
way, or 3 acks or something else.

>>> I'm not sure I like the idea of planned ABI break releases - that strikes
>>> me as a plan where we end up with the same number of ABI breaks as before,
>>> just balled into one release.
>>>
>>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>>> case that each distro will only ship one version of DPDK, or is it possible
>>> that if we have ABI breaks, a distro will provide two copies of DPDK
>>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>>
>>
>> It would probably double validation and maintenance, so it would require
>> a lot of extra effort.
>>
>>>
>>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>>> ABI breaks, and having ABI breaks as a major event reserved only for
>>> massive rework changes, such as major mbuf changes, or new memory layout or
>>> similar.
>>>
>>> Regards,
>>> /Bruce
>>>
>>
> 
> Regards,
> Keith
> 

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 04/04/2019 14:10, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 02:05:27PM +0100, Ray Kinsella wrote:
>>
>>
>> On 04/04/2019 13:02, Luca Boccassi wrote:
>>> On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>
> I would be too, with certain exceptions - rte_mbuf for one. Any structures
> that are used by applications cannot be made opaque, as apps do not want to
> pay the cost of having to call a function every time they want to access
> something from one of those structures.

These the layout of these structures really must become sacrosanct.
As Stephen points out, there may be room for a one more change - fool me
once - to future poof the structure but after, that these structure will
become very hard to change.

> 
> Thankfully for us, we have plenty of other structures that we can work on
> in the meantime that can be made private to avoid ABI breaks! :-) I suggest
> we work through those first, allowing us to hone our ABI-break avoidance
> skills.
> 
> /Bruce
> 

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 04/04/2019 20:08, Wiles, Keith wrote:
> 
> 
>> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>>
>> On 04/04/2019 11:54, Bruce Richardson wrote:
>> <snip>
> ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.

New APIs will be experimental in any case, as you say they are less of
problem.

I agree, if we can make a change and preserve API compatibility with
versioning then everyone is happy.

Changes only need be referred to the higher power in case on absolutely
breakage - _but_ these need to become as rare as hens teeth.

>>
>>> I'm not sure I like the idea of planned ABI break releases - that strikes
>>> me as a plan where we end up with the same number of ABI breaks as before,
>>> just balled into one release.
>>>
>>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>>> case that each distro will only ship one version of DPDK, or is it possible
>>> that if we have ABI breaks, a distro will provide two copies of DPDK
>>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>>
>>
>> It would probably double validation and maintenance, so it would require
>> a lot of extra effort.
>>
>>>
>>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>>> ABI breaks, and having ABI breaks as a major event reserved only for
>>> massive rework changes, such as major mbuf changes, or new memory layout or
>>> similar.
>>>
>>> Regards,
>>> /Bruce
>>>
>>
> 
> Regards,
> Keith
> 

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 04/04/2019 21:13, Kevin Traynor wrote:
> On 04/04/2019 20:08, Wiles, Keith wrote:
>>
>>
>>> On Apr 4, 2019, at 11:56 AM, Kevin Traynor <ktraynor@redhat.com> wrote:
>>>
>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>> <snip>
>>>
>>>>
>>>> My thoughts on the matter are:
>>>> 1. I think we really need to do work to start hiding more of our data
>>>> structures - like what Stephen's latest RFC does. This hiding should reduce
>>>> the scope for ABI breaks.
>>>> 2. Once done, I think we should commit to having an ABI break only in the
>>>> rarest of circumstances, and only with very large justification. I want us
>>>> to get to the point where DPDK releases can immediately be picked up by all
>>>> linux distros and rolled out because they are ABI compatible.
>>>>
>>>
>>> Maybe techboard should explicitly approve ABI breaks and new APIs (or
>>> APIs at transition from experimental to core). Just as a way to get more
>>> eyeballs and scrutiny on them.
>>
>> ABI breaks should be handled by the board. As for new APIs they are not so bad and they do not need to be approved by the board just handled in the normal way. For API changes (I guess that is ABI) needs to be handled by the board unless we use the version control and maintain both APIs for a while.
>>>
> 
> We'll only find out if they are bad when they need ABI breaks later :-)
> 
> My point is a good way to avoid future ABI breaks is to have more
> reviews on the APIs in the first place. Techboard approval might be one
> way, or 3 acks or something else.

+1 on this ... an API break should invite a higher level of scrutiny.

> 
>>>> I'm not sure I like the idea of planned ABI break releases - that strikes
>>>> me as a plan where we end up with the same number of ABI breaks as before,
>>>> just balled into one release.
>>>>
>>>> Question for Kevin, Luca and others who look at distro-packaging: is it the
>>>> case that each distro will only ship one version of DPDK, or is it possible
>>>> that if we have ABI breaks, a distro will provide two copies of DPDK
>>>> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?
>>>>
>>>
>>> It would probably double validation and maintenance, so it would require
>>> a lot of extra effort.
>>>
>>>>
>>>> So, in short, I'm generally in favour of a zero-tolerance approach for DPDK
>>>> ABI breaks, and having ABI breaks as a major event reserved only for
>>>> massive rework changes, such as major mbuf changes, or new memory layout or
>>>> similar.
>>>>
>>>> Regards,
>>>> /Bruce
>>>>
>>>
>>
>> Regards,
>> Keith
>>
> 

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Thomas Monjalon]

05/04/2019 15:25, Ray Kinsella:
> On 04/04/2019 14:10, Bruce Richardson wrote:
> > On Thu, Apr 04, 2019 at 02:05:27PM +0100, Ray Kinsella wrote:
> >> On 04/04/2019 13:02, Luca Boccassi wrote:
> >>> On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> >>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> >>
> > I would be too, with certain exceptions - rte_mbuf for one. Any structures
> > that are used by applications cannot be made opaque, as apps do not want to
> > pay the cost of having to call a function every time they want to access
> > something from one of those structures.
> 
> These the layout of these structures really must become sacrosanct.
> As Stephen points out, there may be room for a one more change - fool me
> once - to future poof the structure but after, that these structure will
> become very hard to change.

Yes, looks like a good plan.

> > Thankfully for us, we have plenty of other structures that we can work on
> > in the meantime that can be made private to avoid ABI breaks! :-) I suggest
> > we work through those first, allowing us to hone our ABI-break avoidance
> > skills.

+1 to work on obvious cases first.

I think we should not wait too much to improve mbuf
with some dynamically registered fields, because it may be
long to achieve such an important feature and make everybody happy.
Note: we may have to sacrifice a few cycles for some features
like distributor or IPsec.

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Thomas Monjalon]

04/04/2019 16:07, Burakov, Anatoly:
> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> > On 04/04/2019 11:54, Bruce Richardson wrote:
> >> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> >> Actually, I think we *do* need to constrain the pace of development for the
> >> sake of ABI stability. At this stage DPDK has been around for quite a
> >> number of years and so should be considered a fairly mature project - it
> >> should just start acting like it.
> > 
> > I 100% agree.
> > 
> > If you break your users stuff regularly enough, they will eventually
> > start looking around for an alternative that doesn't break their stuff
> > quiet so regularly.
> > 
> > We often use the pace of innovation in DPDK as justification for ABI/API
> > breakages, but that approach is a real rarity among the Open Source
> > community. I can't think of any mature project off-hand that share's it.
> > 
> > I would ask is Linux any less innovative because they offer a stable API
> > and have an absolute commitment to never breaking userspace? Would Linux
> > have ever been as popular as it is today it they broke userspace every
> > quarter?
> > 
> > They reality is that they (Linux) find workarounds and compromise
> > because there is an uber-maintainer Linus who had a strong ethos from
> > the start not to break their users stuff - we need the same ethos in DPDK.
> > 
> >>
> >> Now, in terms of features like the memory rework, that is indeed a case
> >> that there was no alternative other than a massive ABI break. However, for
> >> that rework there was a strong need for improvement in that area that we
> >> can make the case for an ABI break to support it - and it is of a scale
> >> that nothing other than an ABI change would do. For other areas and
> >> examples, I doubt there are many in the last couple of years that are of
> >> that scale.
> > 
> > I would also be inclined to agree with Bruce's points on memory rework
> > was somewhat of an outlier, we don't see many like it.
> >> My thoughts on the matter are:
> >> 1. I think we really need to do work to start hiding more of our data
> >> structures - like what Stephen's latest RFC does. This hiding should reduce
> >> the scope for ABI breaks.
> >> 2. Once done, I think we should commit to having an ABI break only in the
> >> rarest of circumstances, and only with very large justification. I want us
> >> to get to the point where DPDK releases can immediately be picked up by all
> >> linux distros and rolled out because they are ABI compatible.
> > 
> > The work that Anatoly describes removing APIs that exposed internal
> > structures and Stephen H's RFC similarly are good examples of the kind
> > of work required to prepare for this change. We need to take a good look
> > at the API and reduce the number of unnecessary internal structures
> > exposed.
> > 
> > I never expected it going to to be a big bang - but is a definite
> > direction we need to move towards over the next few release.
> 
> ...in this case, we have to think long and hard about the fabled EAL 
> rework/split, and in general *specifying* what is it that we want to 
> support, and the use cases that we want to target. Right now there is a 
> huge mountain of technical debt and kludges and workarounds that has 
> accumulated over the years, and it exists precisely because "every 
> change breaks someone's workflow".
> 
> For example, just in memory subsystem alone, we have legacy mem, because 
> some use cases require huge amounts of contiguous memory, and not 
> everyone is using VFIO; there's all of the 32-bit related workarounds 
> and hacks; there's the single-file-segments stuff that could have been 
> the default if not for the fact that we support kernels that don't 
> support fallocate(); there are two different ways of doing in-memory 
> mode, because not all kernels support memfd's; there is a gargantuan 
> pile of workarounds (and "known issues", and just code in general) all 
> over the DPDK codebase just to support our multiprocess model and all of 
> the various warts that come with it.
> 
> In fact, i would even go as far as to say that *most* of EAL ABI breaks 
> have been due to the fact that we store data in shared memory because of 
> multiprocess - so there is simply no way we can change these internal 
> data structures without ABI breaks, because even if they're not exposed 
> through user-facing API, they are still exposed by virtue of secondary 
> processes basically having an ABI contract with primary process instances.
> 
> So, if we are to cement our core API - we have to make a concrete effort 
> to specify what goes and what stays, if we want it to be maintainable. 
> The DPDK 1.0 specification, if you will :)

"DPDK 1.0 specification", that's a great project name :-)

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 07/04/2019 10:48, Thomas Monjalon wrote:
> 04/04/2019 16:07, Burakov, Anatoly:
>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
[SNIP]
>> So, if we are to cement our core API - we have to make a concrete effort 
>> to specify what goes and what stays, if we want it to be maintainable. 
>> The DPDK 1.0 specification, if you will :)
> 
> "DPDK 1.0 specification", that's a great project name :-)
> 

Honestly - I would say that I am nervous of this.

The definition of a DPDK 1.0 specification as a gate to API stability,
feels like a "great plan tomorrow" instead of a "good plan" today. I
think that getting people to dedicate time to such a specification might
prove problematic and I could see this effort being very time consuming.
It might never get completed.

My preference would be to instead adopt a well-publicised community
timeline for adopting more conservative API maintenance rules.

Perhaps we could give ourselves as a community a time-limited window in
which to address concerns around the API before they become hardened -
perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
to 9 months.

We then would know the timeline when niggles like exposure of internal
structures and mbuf structure needed to be sorted by and could
prioritize accordingly.

Ray K

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
> On 07/04/2019 10:48, Thomas Monjalon wrote:
>> 04/04/2019 16:07, Burakov, Anatoly:
>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> [SNIP]
>>> So, if we are to cement our core API - we have to make a concrete effort
>>> to specify what goes and what stays, if we want it to be maintainable.
>>> The DPDK 1.0 specification, if you will :)
>>
>> "DPDK 1.0 specification", that's a great project name :-)
>>
> 
> Honestly - I would say that I am nervous of this.
> 
> The definition of a DPDK 1.0 specification as a gate to API stability,
> feels like a "great plan tomorrow" instead of a "good plan" today. I
> think that getting people to dedicate time to such a specification might
> prove problematic and I could see this effort being very time consuming.
> It might never get completed.
> 
> My preference would be to instead adopt a well-publicised community
> timeline for adopting more conservative API maintenance rules.
> 
> Perhaps we could give ourselves as a community a time-limited window in
> which to address concerns around the API before they become hardened -
> perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
> to 9 months.
> 
> We then would know the timeline when niggles like exposure of internal
> structures and mbuf structure needed to be sorted by and could
> prioritize accordingly.
> 
> Ray K
> 

My worry here is that some API's get more attention than others, but 
requirements for freezing the API/ABI are applicable to all of them.

Everyone loves discussing specifics of mbufs and dev API's, and I have 
no doubt that DPDK community can arrive at a consensus with regards to 
mbuf format etc. in a timely manner, since everyone has a vested 
interest in those covering their use cases. I have way less confidence 
in us possibly having saner and more maintainable platform 
initialization code, simply because any attempt to change those will 
likely be met with "please keep all of the old stuff working", which 
gets us right back to where we started.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 08/04/2019 11:15, Burakov, Anatoly wrote:
> On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
>> On 07/04/2019 10:48, Thomas Monjalon wrote:
>>> 04/04/2019 16:07, Burakov, Anatoly:
>>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>> [SNIP]
[SNIP]
> 
> My worry here is that some API's get more attention than others, but
> requirements for freezing the API/ABI are applicable to all of them.
> 
> Everyone loves discussing specifics of mbufs and dev API's, and I have
> no doubt that DPDK community can arrive at a consensus with regards to
> mbuf format etc. in a timely manner, since everyone has a vested
> interest in those covering their use cases. 
> I have way less confidence
> in us possibly having saner and more maintainable platform
> initialization code, 

I think you are right, however its that same lack of enthusiasm that
would hamper a DPDK 1.0 specification. Similarly I can see a DPDK 1.0
specification effort becoming an endless process, as it would always be
low on DPDK contributors priority list.

Instead I recommend we set an API freeze date to focus peoples minds,
advertise what it means and contributors will respond and look after the
areas they care/are responsible for.

> simply because any attempt to change those will
> likely be met with "please keep all of the old stuff working", which
> gets us right back to where we started.
> 

And to be honest that is a fair enough expectation from them, right?
To Stephen's if we need to break it, let's do it once more and then
never again.

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 08-Apr-19 2:00 PM, Ray Kinsella wrote:
> 
> 
> On 08/04/2019 11:15, Burakov, Anatoly wrote:
>> On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
>>> On 07/04/2019 10:48, Thomas Monjalon wrote:
>>>> 04/04/2019 16:07, Burakov, Anatoly:
>>>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>> [SNIP]
> [SNIP]
>>
>> My worry here is that some API's get more attention than others, but
>> requirements for freezing the API/ABI are applicable to all of them.
>>
>> Everyone loves discussing specifics of mbufs and dev API's, and I have
>> no doubt that DPDK community can arrive at a consensus with regards to
>> mbuf format etc. in a timely manner, since everyone has a vested
>> interest in those covering their use cases.
>> I have way less confidence
>> in us possibly having saner and more maintainable platform
>> initialization code,
> 
> I think you are right, however its that same lack of enthusiasm that
> would hamper a DPDK 1.0 specification. Similarly I can see a DPDK 1.0
> specification effort becoming an endless process, as it would always be
> low on DPDK contributors priority list.
> 
> Instead I recommend we set an API freeze date to focus peoples minds,
> advertise what it means and contributors will respond and look after the
> areas they care/are responsible for.
> 
>> simply because any attempt to change those will
>> likely be met with "please keep all of the old stuff working", which
>> gets us right back to where we started.
>>
> 
> And to be honest that is a fair enough expectation from them, right?
> To Stephen's if we need to break it, let's do it once more and then
> never again.
> 

Perhaps i didn't phrase myself well enough. When i say "DPDK 1.0 
specification" i don't mean there literally should be a document with a 
formal specification :)

Rather, i'm thinking more along the lines of, let's take a look at what 
we have, and think of ways we can reduce the amount of code and improve 
things without causing major inconveniences to great majority of users. 
As in, if we want to "break once more and then never again", then maybe 
instead of small incremental fixes here and there, we could actually do 
a more drastic change that keeps perhaps 90% users happy instead of 
100%, but which reduces maintenance/validation effort from 100% down to 30%.

As a concrete proposal, my number one dream would be to see multiprocess 
gone. I also recall desire for "DPDK to be more lightweight", and i 
maintain that DPDK *cannot* be lightweight if we are to support 
multiprocess - we can have one or the other, but not both. However, 
realistically, i don't think dropping multiprocess is ever going to 
happen - not only it is too entrenched in DPDK use cases, it is actually 
quite useful despite its flaws.

However, what *could* be realistic is to trim down complexity in EAL 
init and system code in general - to me (again, a concrete proposal!), 
that would be dropping igb_uio and supporting upstream kernel modules 
only (VFIO, maybe uio_pci_generic if we're pushing it?), dropping 32-bit 
support and legacy mem modes, and perhaps bumping kernel version and 
removing some compatibility code as well. This would remove potentially 
thousands of lines of code and keep EAL cleaner and more maintainable: 
right now, we have two different hotplug mechanisms (UIO and VFIO), lots 
of different memory/interrupt modes, etc., for no reason that is readily 
apparent to me.

So, as you can see, an effort to review and delineate what we want to 
support would be necessary if we want to ease the maintenance burden for 
either myself or anyone that inherits this code, as well as reducing the 
number of moving parts and, as a consequence, validation effort and 
amount of bugs. We can't simply do it in a random release "just 
because", but if we were to "break once more and then never again", 
perhaps this is something that could be considered.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[David Marchand]

On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> As a concrete proposal, my number one dream would be to see multiprocess
> gone. I also recall desire for "DPDK to be more lightweight", and i
> maintain that DPDK *cannot* be lightweight if we are to support
> multiprocess - we can have one or the other, but not both. However,
> realistically, i don't think dropping multiprocess is ever going to
> happen - not only it is too entrenched in DPDK use cases, it is actually
> quite useful despite its flaws.
>

Well, honestly, I'd like to hear about this.
What are the real usecases for multi process support?
Do we have even a single opensource project that uses it?


-- 
David Marchand

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 08-Apr-19 2:58 PM, David Marchand wrote:
> On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     As a concrete proposal, my number one dream would be to see
>     multiprocess
>     gone. I also recall desire for "DPDK to be more lightweight", and i
>     maintain that DPDK *cannot* be lightweight if we are to support
>     multiprocess - we can have one or the other, but not both. However,
>     realistically, i don't think dropping multiprocess is ever going to
>     happen - not only it is too entrenched in DPDK use cases, it is
>     actually
>     quite useful despite its flaws.
> 
> 
> Well, honestly, I'd like to hear about this.
> What are the real usecases for multi process support?
> Do we have even a single opensource project that uses it?
> 

I'm aware of a few closed source usages of multiprocess. I also think 
current versions of collectd rely on secondary process (there's been a 
Telemetry API added to avoid that, but AFAIK the support for Telemetry 
is not upstream in collectd yet), and so do/would any dump-style 
applications - in fact, we ourselves include one such application in our 
codebase (pdump, proc-info, etc.).

> 
> -- 
> David Marchand


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[David Marchand]

On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 08-Apr-19 2:58 PM, David Marchand wrote:
> > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> >
> >     As a concrete proposal, my number one dream would be to see
> >     multiprocess
> >     gone. I also recall desire for "DPDK to be more lightweight", and i
> >     maintain that DPDK *cannot* be lightweight if we are to support
> >     multiprocess - we can have one or the other, but not both. However,
> >     realistically, i don't think dropping multiprocess is ever going to
> >     happen - not only it is too entrenched in DPDK use cases, it is
> >     actually
> >     quite useful despite its flaws.
> >
> >
> > Well, honestly, I'd like to hear about this.
> > What are the real usecases for multi process support?
> > Do we have even a single opensource project that uses it?
> >
>
> I'm aware of a few closed source usages of multiprocess. I also think
> current versions of collectd rely on secondary process (there's been a
> Telemetry API added to avoid that, but AFAIK the support for Telemetry
> is not upstream in collectd yet), and so do/would any dump-style
> applications - in fact, we ourselves include one such application in our
> codebase (pdump, proc-info, etc.).
>

Sorry, I don't want to highjack this thread, I can start a separate thread
if people feel like it.
If we go with stabilisation, we must be careful that we want to support the
features.

So about multiprocess, again, in those closed source projects you know of,
what are the usecases?

For what we provide in dpdk pdump, proc-info, referring to oneself is not
that convincing to me as I don't use those tools.

I don't see what we could not achieve the same with a control thread
running in the dpdk process and handling commands.
It would be open to the outside via a more standard channel, like a UNIX
socket or something like this.
If we need to declare a dynamic channel, it can be constructed as an
extension of the existing standard channel: we can open something like a
POSIX shm and push things in it.
Was this explored ?


-- 
David Marchand

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Stephen Hemminger]

On Mon, 8 Apr 2019 16:38:55 +0200
David Marchand <david.marchand@redhat.com> wrote:

> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly <anatoly.burakov@intel.com>
> wrote:
> 
> > On 08-Apr-19 2:58 PM, David Marchand wrote:  
> > > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> > >
> > >     As a concrete proposal, my number one dream would be to see
> > >     multiprocess
> > >     gone. I also recall desire for "DPDK to be more lightweight", and i
> > >     maintain that DPDK *cannot* be lightweight if we are to support
> > >     multiprocess - we can have one or the other, but not both. However,
> > >     realistically, i don't think dropping multiprocess is ever going to
> > >     happen - not only it is too entrenched in DPDK use cases, it is
> > >     actually
> > >     quite useful despite its flaws.
> > >
> > >
> > > Well, honestly, I'd like to hear about this.
> > > What are the real usecases for multi process support?
> > > Do we have even a single opensource project that uses it?
> > >  
> >
> > I'm aware of a few closed source usages of multiprocess. I also think
> > current versions of collectd rely on secondary process (there's been a
> > Telemetry API added to avoid that, but AFAIK the support for Telemetry
> > is not upstream in collectd yet), and so do/would any dump-style
> > applications - in fact, we ourselves include one such application in our
> > codebase (pdump, proc-info, etc.).
> >  
> 
> Sorry, I don't want to highjack this thread, I can start a separate thread
> if people feel like it.
> If we go with stabilisation, we must be careful that we want to support the
> features.
> 
> So about multiprocess, again, in those closed source projects you know of,
> what are the usecases?
> 
> For what we provide in dpdk pdump, proc-info, referring to oneself is not
> that convincing to me as I don't use those tools.
> 
> I don't see what we could not achieve the same with a control thread
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an
> extension of the existing standard channel: we can open something like a
> POSIX shm and push things in it.
> Was this explored ?
> 
> 

Yes, there are several closed source applications using multi-process.
But the problem with that is no one tests with all the drivers, api's and
configurations in DPDK.

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 08-Apr-19 3:38 PM, David Marchand wrote:
> 
> 
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 08-Apr-19 2:58 PM, David Marchand wrote:
>      > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
>      > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
>     <mailto:anatoly.burakov@intel.com
>     <mailto:anatoly.burakov@intel.com>>> wrote:
>      >
>      >     As a concrete proposal, my number one dream would be to see
>      >     multiprocess
>      >     gone. I also recall desire for "DPDK to be more lightweight",
>     and i
>      >     maintain that DPDK *cannot* be lightweight if we are to support
>      >     multiprocess - we can have one or the other, but not both.
>     However,
>      >     realistically, i don't think dropping multiprocess is ever
>     going to
>      >     happen - not only it is too entrenched in DPDK use cases, it is
>      >     actually
>      >     quite useful despite its flaws.
>      >
>      >
>      > Well, honestly, I'd like to hear about this.
>      > What are the real usecases for multi process support?
>      > Do we have even a single opensource project that uses it?
>      >
> 
>     I'm aware of a few closed source usages of multiprocess. I also think
>     current versions of collectd rely on secondary process (there's been a
>     Telemetry API added to avoid that, but AFAIK the support for Telemetry
>     is not upstream in collectd yet), and so do/would any dump-style
>     applications - in fact, we ourselves include one such application in
>     our
>     codebase (pdump, proc-info, etc.).
> 
> 
> Sorry, I don't want to highjack this thread, I can start a separate 
> thread if people feel like it.
> If we go with stabilisation, we must be careful that we want to support 
> the features.
> 
> So about multiprocess, again, in those closed source projects you know 
> of, what are the usecases?
> 
> For what we provide in dpdk pdump, proc-info, referring to oneself is 
> not that convincing to me as I don't use those tools.
> 
> I don't see what we could not achieve the same with a control thread 
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX 
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an 
> extension of the existing standard channel: we can open something like a 
> POSIX shm and push things in it.
> Was this explored ?

There are certainly things that we can do that can make some aspects of 
multiprocess redundant. For example, for any kind of collectd-like 
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably 
provide a better and more maintainable way of doing things.

Our multiprocess also makes it easier to write pipeline/load-balancing 
type applications. To see an example, look at our 
multiprocess/client-server example. This is demonstrating how, instead 
of writing one big monolithic application, one could instead write a 
number of smaller applications each doing their thing. It is of course 
possible to do the same without multiprocess, as evidenced by our sample 
applications such as load-balancer, distributor, ip-pipeline etc., but 
it is arguably easier to implement *real* applications that way due to 
separation of concerns and more focused codebase.

However, there are two use cases that i can think of that are either 
hard or outright not possible without our multiprocess API's. The first 
one is dumping functionality. For example, dpdk_proc_info can display 
info from a currently-running or defunct process - list its 
memzones/mempools/etc. - basically, everything there is to know about 
the shared memory can be known that way. While this isn't a "real" use 
case, it is useful for debugging.

More importantly, our multiprocess model provides resilience. In an 
event of a crash, the entire application is not brought down - instead, 
only the crashed process goes down. It's not /perfect/ resilience, of 
course, and there are caveats (memory leaking, locks, etc.), but you do 
get /some/ resilience that way - your process went down, you spin 
another secondary and you're back up and running again.

The above described scenario is how most people (that i know of) appear 
to be using multiprocess - some kind of "crash-resilient" 
load-balancing/pipelining app.

> 
> 
> -- 
> David Marchand


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Burakov, Anatoly]

On 08-Apr-19 3:38 PM, David Marchand wrote:
> 
> 
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 08-Apr-19 2:58 PM, David Marchand wrote:
>      > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
>      > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
>     <mailto:anatoly.burakov@intel.com
>     <mailto:anatoly.burakov@intel.com>>> wrote:
>      >
>      >     As a concrete proposal, my number one dream would be to see
>      >     multiprocess
>      >     gone. I also recall desire for "DPDK to be more lightweight",
>     and i
>      >     maintain that DPDK *cannot* be lightweight if we are to support
>      >     multiprocess - we can have one or the other, but not both.
>     However,
>      >     realistically, i don't think dropping multiprocess is ever
>     going to
>      >     happen - not only it is too entrenched in DPDK use cases, it is
>      >     actually
>      >     quite useful despite its flaws.
>      >
>      >
>      > Well, honestly, I'd like to hear about this.
>      > What are the real usecases for multi process support?
>      > Do we have even a single opensource project that uses it?
>      >
> 
>     I'm aware of a few closed source usages of multiprocess. I also think
>     current versions of collectd rely on secondary process (there's been a
>     Telemetry API added to avoid that, but AFAIK the support for Telemetry
>     is not upstream in collectd yet), and so do/would any dump-style
>     applications - in fact, we ourselves include one such application in
>     our
>     codebase (pdump, proc-info, etc.).
> 
> 
> Sorry, I don't want to highjack this thread, I can start a separate 
> thread if people feel like it.
> If we go with stabilisation, we must be careful that we want to support 
> the features.
> 
> So about multiprocess, again, in those closed source projects you know 
> of, what are the usecases?
> 
> For what we provide in dpdk pdump, proc-info, referring to oneself is 
> not that convincing to me as I don't use those tools.
> 
> I don't see what we could not achieve the same with a control thread 
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX 
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an 
> extension of the existing standard channel: we can open something like a 
> POSIX shm and push things in it.
> Was this explored ?

There are certainly things that we can do that can make some aspects of 
multiprocess redundant. For example, for any kind of collectd-like 
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably 
provide a better and more maintainable way of doing things.

Our multiprocess also makes it easier to write pipeline/load-balancing 
type applications. To see an example, look at our 
multiprocess/client-server example. This is demonstrating how, instead 
of writing one big monolithic application, one could instead write a 
number of smaller applications each doing their thing. It is of course 
possible to do the same without multiprocess, as evidenced by our sample 
applications such as load-balancer, distributor, ip-pipeline etc., but 
it is arguably easier to implement *real* applications that way due to 
separation of concerns and more focused codebase.

However, there are two use cases that i can think of that are either 
hard or outright not possible without our multiprocess API's. The first 
one is dumping functionality. For example, dpdk_proc_info can display 
info from a currently-running or defunct process - list its 
memzones/mempools/etc. - basically, everything there is to know about 
the shared memory can be known that way. While this isn't a "real" use 
case, it is useful for debugging.

More importantly, our multiprocess model provides resilience. In an 
event of a crash, the entire application is not brought down - instead, 
only the crashed process goes down. It's not /perfect/ resilience, of 
course, and there are caveats (memory leaking, locks, etc.), but you do 
get /some/ resilience that way - your process went down, you spin 
another secondary and you're back up and running again.

The above described scenario is how most people (that i know of) appear 
to be using multiprocess - some kind of "crash-resilient" 
load-balancing/pipelining app.

> 
> 
> -- 
> David Marchand


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Ray Kinsella]

On 08/04/2019 14:38, Burakov, Anatoly wrote:
> On 08-Apr-19 2:00 PM, Ray Kinsella wrote:
>>
>>
>> On 08/04/2019 11:15, Burakov, Anatoly wrote:
>>> On 08-Apr-19 10:04 AM, Ray Kinsella wrote:
>>>> On 07/04/2019 10:48, Thomas Monjalon wrote:
>>>>> 04/04/2019 16:07, Burakov, Anatoly:
>>>>>> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
>>>>>>> On 04/04/2019 11:54, Bruce Richardson wrote:
>>>>>>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
>>>>>>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
>>>> [SNIP]
>> [SNIP]
[SNIP]
>>
> 
> Perhaps i didn't phrase myself well enough. When i say "DPDK 1.0
> specification" i don't mean there literally should be a document with a
> formal specification :)

When I hear the word specification I think encyclopedia-like stacks of
paper.

> 
> Rather, i'm thinking more along the lines of, let's take a look at what
> we have, and think of ways we can reduce the amount of code and improve
> things without causing major inconveniences to great majority of users.
> As in, if we want to "break once more and then never again", then maybe
> instead of small incremental fixes here and there, we could actually do
> a more drastic change that keeps perhaps 90% users happy instead of
> 100%, but which reduces maintenance/validation effort from 100% down to
> 30%.

Agreed - all I would propose on top of this is that we give ourselves a
real deadline.

The DPDK API has been iterating since 2012 and while we can spare a few
more months to do tidy-up and get it right, we _do_ need to draw a line
in the sand at the same time.

> 
> As a concrete proposal, my number one dream would be to see multiprocess
> gone. I also recall desire for "DPDK to be more lightweight", and i
> maintain that DPDK *cannot* be lightweight if we are to support
> multiprocess - we can have one or the other, but not both. However,
> realistically, i don't think dropping multiprocess is ever going to
> happen - not only it is too entrenched in DPDK use cases, it is actually
> quite useful despite its flaws.
> 
> However, what *could* be realistic is to trim down complexity in EAL
> init and system code in general - to me (again, a concrete proposal!),
> that would be dropping igb_uio and supporting upstream kernel modules
> only (VFIO, maybe uio_pci_generic if we're pushing it?), dropping 32-bit
> support and legacy mem modes, and perhaps bumping kernel version and
> removing some compatibility code as well. This would remove potentially
> thousands of lines of code and keep EAL cleaner and more maintainable:
> right now, we have two different hotplug mechanisms (UIO and VFIO), lots
> of different memory/interrupt modes, etc., for no reason that is readily
> apparent to me.
> 
> So, as you can see, an effort to review and delineate what we want to
> support would be necessary if we want to ease the maintenance burden for
> either myself or anyone that inherits this code, as well as reducing the
> number of moving parts and, as a consequence, validation effort and
> amount of bugs. We can't simply do it in a random release "just
> because", but if we were to "break once more and then never again",
> perhaps this is something that could be considered.
> 

Re: [dpdk-dev] DPDK ABI/API Stability

[Honnappa Nagarahalli]

> 
> Hi folks,
> 
> Recently I started a discussion with the DPDK Technical Board on DPDK
> ABI/API stability. This was born out informal feedback I had received from
> a number of users of DPDK about ABI churn. In turn this feedback then
> prompted an ABI analysis of DPDK using tools from abi-laboratory.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
This link can be used to check which structures are not needed to be exposed to the application.
For ex: for rte_hash library in "19.02 vs current", backward compatibility is ~69%. Digging into further details indicates that the break is due to addition of a member to, what I think should be, an internal structure (but is external).

> 
> I guess the short story is that DPDK ABI hasn't really settled down as the
> project has matured. If you take a look at the “Backward Compat.”
> column which measures ABI compatibility compared to the previous
> releases, you will see significant churn in the ABI over successive releases
> since v16.04.
> 
> Now compare DPDK to GStreamer as an example of a very mature project
> with a similar intent, a framework for building applications, and which
> enjoys a very stable API.
> 
> https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> 
> The DPDK ABI churn has the following affects for users:-
> 
> 1. The churn obliges users of DPDK to commit to a constant re-integration
> and re-validation effort for new versions of DPDK. This effort from their
> perspective may not add value to their consuming project, particular if
> they are only updating to "stay current".
Even if the ABI did not change, any claim of support with newer version needs re-validation. I think, re-integration is the only extra effort.
Why would anyone like to move to newer version just to stay current if the newer version does not add any value to them? IMO, this is doing work for no benefit.

> 2. The churn encourages users of DPDK to slip versions, putting off
> reintegration to later, building up technical debt and causing their projects
> to miss support for new hardware or features.
> 3. It makes DPDK different to almost every other system library and
> framework that an operating systems might ship. This makes DPDK trickier
> to dynamically link against, package and maintain for OS maintainers.
> 
> In order to address this issue, I have put together the minimal set of
> concrete proposals below for discussion at the Technical Board next
> Wednesday.
> 
> I wanted to share this, as these might not yet be the right proposals,
> however I am putting them out there for feedback to start the discussion.
> 
> Thanks,
> 
> Ray K
> 
> 
> Experimental API
> 1.	APIs designated as experimental are not considered part of the ABI
> and may change without warning at any time.
> 2.	APIs designated as experimental must be marked depreciated for a
> least one quarterly release before removal.
> 3.	APIs designated as experimental will no longer automatically
> graduate
> to core after one release, they may stay experimental until their author
> and the maintainer agree that graduation is appropriate.
> 
> Core API (non-experimental API)
> 4.	APIs designated as core must be depreciated for a least two years
> before removal, to facilitate the continued compatibility with LTS releases.
> A final removal notice will be published to the DPDK Mailing List, and if
> there are no strong objections only then an API may be removed.
> 5.	APIs designated as core may be changed as follows:-
> 5.a	The change proposer must demonstrated that the change has a
> supporting use case and could not be achieved in any other way.
> 5.b	ABI version compatibility must be retained, as described below.
> 
> Shared Libraries
> 6.	DPDK will move to shared libraries & dynamic linking by default, to
> accommodate greater use of ABI versioning by DPDK consumers.
+1, however, I think applications should have been doing this already (if they were bothered with ABI compatibility)
Would not https://doc.dpdk.org/guides/contributing/versioning.html#abi-versioning solve the issue? Are we not doing versioning for some changes?

<snip>

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[David Marchand]

On Mon, Apr 8, 2019 at 5:52 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

However, there are two use cases that i can think of that are either
> hard or outright not possible without our multiprocess API's. The first
> one is dumping functionality. For example, dpdk_proc_info can display
> info from a currently-running or defunct process - list its
> memzones/mempools/etc. - basically, everything there is to know about
> the shared memory can be known that way. While this isn't a "real" use
> case, it is useful for debugging.
>

On the principle (there might be some corner cases I don't have an idea of)
attaching an existing process is something that can be done with PTRACE_*.
As for dumping post mortem, again, this could be done without dpdk assist.
The only thing that should come from dpdk is the knowledge of where and how
to dump the objects.



> More importantly, our multiprocess model provides resilience. In an
> event of a crash, the entire application is not brought down - instead,
> only the crashed process goes down. It's not /perfect/ resilience, of
> course, and there are caveats (memory leaking, locks, etc.), but you do
> get /some/ resilience that way - your process went down, you spin
> another secondary and you're back up and running again.
>

Never witnessed a recover, and yes I would suspect quite some funny
situations like locks, dirty memory etc...
Okay, thanks for the inputs.


-- 
David Marchand

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Bruce Richardson]

On Wed, Apr 10, 2019 at 05:14:43AM +0000, Honnappa Nagarahalli wrote:
> > 
> > Hi folks,
> > 
> > Recently I started a discussion with the DPDK Technical Board on DPDK
> > ABI/API stability. This was born out informal feedback I had received from
> > a number of users of DPDK about ABI churn. In turn this feedback then
> > prompted an ABI analysis of DPDK using tools from abi-laboratory.
> > 
> > https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> This link can be used to check which structures are not needed to be exposed to the application.
> For ex: for rte_hash library in "19.02 vs current", backward compatibility is ~69%. Digging into further details indicates that the break is due to addition of a member to, what I think should be, an internal structure (but is external).
> 
> > 
> > I guess the short story is that DPDK ABI hasn't really settled down as the
> > project has matured. If you take a look at the “Backward Compat.”
> > column which measures ABI compatibility compared to the previous
> > releases, you will see significant churn in the ABI over successive releases
> > since v16.04.
> > 
> > Now compare DPDK to GStreamer as an example of a very mature project
> > with a similar intent, a framework for building applications, and which
> > enjoys a very stable API.
> > 
> > https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> > 
> > The DPDK ABI churn has the following affects for users:-
> > 
> > 1. The churn obliges users of DPDK to commit to a constant re-integration
> > and re-validation effort for new versions of DPDK. This effort from their
> > perspective may not add value to their consuming project, particular if
> > they are only updating to "stay current".
> Even if the ABI did not change, any claim of support with newer version needs re-validation. I think, re-integration is the only extra effort.
> Why would anyone like to move to newer version just to stay current if the newer version does not add any value to them? IMO, this is doing work for no benefit.
> 
> > 2. The churn encourages users of DPDK to slip versions, putting off
> > reintegration to later, building up technical debt and causing their projects
> > to miss support for new hardware or features.
> > 3. It makes DPDK different to almost every other system library and
> > framework that an operating systems might ship. This makes DPDK trickier
> > to dynamically link against, package and maintain for OS maintainers.
> > 
> > In order to address this issue, I have put together the minimal set of
> > concrete proposals below for discussion at the Technical Board next
> > Wednesday.
> > 
> > I wanted to share this, as these might not yet be the right proposals,
> > however I am putting them out there for feedback to start the discussion.
> > 
> > Thanks,
> > 
> > Ray K
> > 
> > 
> > Experimental API
> > 1.	APIs designated as experimental are not considered part of the ABI
> > and may change without warning at any time.
> > 2.	APIs designated as experimental must be marked depreciated for a
> > least one quarterly release before removal.
> > 3.	APIs designated as experimental will no longer automatically
> > graduate
> > to core after one release, they may stay experimental until their author
> > and the maintainer agree that graduation is appropriate.
> > 
> > Core API (non-experimental API)
> > 4.	APIs designated as core must be depreciated for a least two years
> > before removal, to facilitate the continued compatibility with LTS releases.
> > A final removal notice will be published to the DPDK Mailing List, and if
> > there are no strong objections only then an API may be removed.
> > 5.	APIs designated as core may be changed as follows:-
> > 5.a	The change proposer must demonstrated that the change has a
> > supporting use case and could not be achieved in any other way.
> > 5.b	ABI version compatibility must be retained, as described below.
> > 
> > Shared Libraries
> > 6.	DPDK will move to shared libraries & dynamic linking by default, to
> > accommodate greater use of ABI versioning by DPDK consumers.
> +1, however, I think applications should have been doing this already (if they were bothered with ABI compatibility)
> Would not https://doc.dpdk.org/guides/contributing/versioning.html#abi-versioning solve the issue? Are we not doing versioning for some changes?
> 
We do sometimes, but not enough. If we mandated ABI compatibility, or made
ABI breaks harder, I think we'd see greater adoption of this versioning
(something I am very much favour of).

/Bruce

Re: [dpdk-dev] DPDK ABI/API Stability

[Luca Boccassi]

On Wed, 2019-04-10 at 05:14 +0000, Honnappa Nagarahalli wrote:
> > I guess the short story is that DPDK ABI hasn't really settled down
> > as the
> > project has matured. If you take a look at the “Backward Compat.”
> > column which measures ABI compatibility compared to the previous
> > releases, you will see significant churn in the ABI over successive
> > releases
> > since v16.04.
> > 
> > Now compare DPDK to GStreamer as an example of a very mature
> > project
> > with a similar intent, a framework for building applications, and
> > which
> > enjoys a very stable API.
> > 
> > https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> > 
> > 
> > The DPDK ABI churn has the following affects for users:-
> > 
> > 1. The churn obliges users of DPDK to commit to a constant re-
> > integration
> > and re-validation effort for new versions of DPDK. This effort from
> > their
> > perspective may not add value to their consuming project,
> > particular if
> > they are only updating to "stay current".
> 
> Even if the ABI did not change, any claim of support with newer
> version needs re-validation. I think, re-integration is the only
> extra effort.

From first-hand experience: re-integration and re-validation are
different in scope and resource requirements.
Validation is usually done by the QA group, and usually doesn't cover
just one library that makes up a part of a product. In other words,
whether the DPDK version changes or not, a new version of a product
will typically undergo full regression testing anyway.

Integration is done by the development group. Any engineering-days that
have to be dedicated to re-integrate a new version of DPDK are
resources taken away from development of new features or bug fixing.
Maintenance costs of OSS components are a sometimes overlooked but
critical part of correctly scoping the opex of a project, and it's
something that product/project managers do look at.

> Why would anyone like to move to newer version just to stay current
> if the newer version does not add any value to them? IMO, this is
> doing work for no benefit.

For many reasons. For example the argument I always use is that while
new version Y might not be needed, new version Z might suddenly become
required for the successful delivery of critical feature A, or to fix
critical bug B.
In my experience, jumping from version X to Y and then Y to Z is always
cheaper and quicker and lower effort than jumping from X to Z, and the
larger the jump, the more work it is.

Another reason is that it's orders of magnitude cheaper to consume
dependencies from the base distribution of choice when building a Linux
product, rather than vendorizing. Doing that has of course drawbacks,
including not being in control of the version of dependencies - so you
don't have a choice, you need to keep up as the base distro moves on.

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Neil Horman]

On Mon, Apr 08, 2019 at 10:04:21AM +0100, Ray Kinsella wrote:
> On 07/04/2019 10:48, Thomas Monjalon wrote:
> > 04/04/2019 16:07, Burakov, Anatoly:
> >> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> >>> On 04/04/2019 11:54, Bruce Richardson wrote:
> >>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> >>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> [SNIP]
> >> So, if we are to cement our core API - we have to make a concrete effort 
> >> to specify what goes and what stays, if we want it to be maintainable. 
> >> The DPDK 1.0 specification, if you will :)
> > 
> > "DPDK 1.0 specification", that's a great project name :-)
> > 
> 
> Honestly - I would say that I am nervous of this.
> 
> The definition of a DPDK 1.0 specification as a gate to API stability,
> feels like a "great plan tomorrow" instead of a "good plan" today. I
> think that getting people to dedicate time to such a specification might
> prove problematic and I could see this effort being very time consuming.
> It might never get completed.
> 
> My preference would be to instead adopt a well-publicised community
> timeline for adopting more conservative API maintenance rules.
> 
> Perhaps we could give ourselves as a community a time-limited window in
> which to address concerns around the API before they become hardened -
> perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
> to 9 months.
> 
> We then would know the timeline when niggles like exposure of internal
> structures and mbuf structure needed to be sorted by and could
> prioritize accordingly.
> 
> Ray K

I'm hesitant to say this, because I'm not usually a fan of throwing up
barricades to progress, but might some level of CI integration be useful here?

Part of the problem, as I've seen it (and I think you've noted previously in
this thread), is that ABI stability just hasn't been a priority, and not
something that developers look at when making changes, nor when reviewers review
patches.  When I wrote the early ABI checking tools for DPDK, while the reaction
was generally positive (I think), the results were informational, and treated as
such (something to take note of perhaps, but something that could be ignored if
there were more pressing issues).  Perhaps a concrete step might be to run the
ABI checker during a CI run on every commit, and block acceptance of a patch if
it modifies the ABI.  That would at least put a procedural break in ABI
modification without clear approval from the board.

Neil

Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Bruce Richardson]

On Sat, Apr 13, 2019 at 08:42:02PM -0400, Neil Horman wrote:
> On Mon, Apr 08, 2019 at 10:04:21AM +0100, Ray Kinsella wrote:
> > On 07/04/2019 10:48, Thomas Monjalon wrote:
> > > 04/04/2019 16:07, Burakov, Anatoly:
> > >> On 04-Apr-19 1:52 PM, Ray Kinsella wrote:
> > >>> On 04/04/2019 11:54, Bruce Richardson wrote:
> > >>>> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> > >>>>> On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> > [SNIP]
> > >> So, if we are to cement our core API - we have to make a concrete effort 
> > >> to specify what goes and what stays, if we want it to be maintainable. 
> > >> The DPDK 1.0 specification, if you will :)
> > > 
> > > "DPDK 1.0 specification", that's a great project name :-)
> > > 
> > 
> > Honestly - I would say that I am nervous of this.
> > 
> > The definition of a DPDK 1.0 specification as a gate to API stability,
> > feels like a "great plan tomorrow" instead of a "good plan" today. I
> > think that getting people to dedicate time to such a specification might
> > prove problematic and I could see this effort being very time consuming.
> > It might never get completed.
> > 
> > My preference would be to instead adopt a well-publicised community
> > timeline for adopting more conservative API maintenance rules.
> > 
> > Perhaps we could give ourselves as a community a time-limited window in
> > which to address concerns around the API before they become hardened -
> > perhaps say until DPDK 19.11 LTS, or something of the order of 6 months
> > to 9 months.
> > 
> > We then would know the timeline when niggles like exposure of internal
> > structures and mbuf structure needed to be sorted by and could
> > prioritize accordingly.
> > 
> > Ray K
> 
> I'm hesitant to say this, because I'm not usually a fan of throwing up
> barricades to progress, but might some level of CI integration be useful here?
> 
> Part of the problem, as I've seen it (and I think you've noted previously in
> this thread), is that ABI stability just hasn't been a priority, and not
> something that developers look at when making changes, nor when reviewers review
> patches.  When I wrote the early ABI checking tools for DPDK, while the reaction
> was generally positive (I think), the results were informational, and treated as
> such (something to take note of perhaps, but something that could be ignored if
> there were more pressing issues).  Perhaps a concrete step might be to run the
> ABI checker during a CI run on every commit, and block acceptance of a patch if
> it modifies the ABI.  That would at least put a procedural break in ABI
> modification without clear approval from the board.
> 
No objections to that here. Sounds a reasonable suggestion.