kunit: what do we do with the 'kunit/alpha/master' branch?

kunit: what do we do with the 'kunit/alpha/master' branch?

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 8886 bytes --]

TL;DR: We expect KUnit to be accepted upstream into Linus' branch in
the next week or two, and we now need to figure out what we are going
to do with our non-upstream 'kunit/alpha/master' branch.


Hello everyone,

We've put together a rough proposal of what we should do with our
not-upstream branch, known to most people using it as
'kunit/alpha/master'[1], now that KUnit's acceptance into mainline
appears to be imminent (the KUnit MVP patchset is now in linux-next,
and the merge window just opened).

==========
Background
==========

KUnit development is currently split between two versions: the
'kunit/alpha/master'[1] git branch, and the version being submitted to
the upstream Linux kernel. While there are some good reasons to
continue to have two separate versions of KUnit, at present there is
some uncertainty around the difference between these versions, and in
which circumstances each version is useful.

At present, the 'kunit/alpha/master' branch serves a few different
purposes. It is a place for code not-yet-ready for upstream -- such as
the mocking framework -- while being developed, while also acting as a
stable version for customers who do not wish to follow along with the
changes made during the upstreaming process. Adding to the confusion,
the name 'kunit/alpha/master' refers to an early (alpha) version of
KUnit, and the version of KUnit being upstreamed has now diverged
significantly from this version, requiring significant differences in
documentation, and requiring a number of changes to tests when porting
from one version to the other. Finally, it is not clear how the
'kunit/alpha/master' version should evolve as features it contains are
upstreamed.

On the other hand, the version being upstreamed has its own
complications. It contains significantly fewer features (as features
such as the mocking frameworks will be upstreamed individually), and
so is less useful for the average customer. Until each feature is
upstreamed, it is iterated on rapidly to address comments from the
kernel community, so in-progress features are not stable enough to
reasonably build on. Finally, it exists only as a set of patches on
mailing lists, rather than as a maintained git repository (due to the
fact that the patches themselves are changing rapidly), making it
difficult for early adopters to incorporate into their own trees.

Whilst we believe there to be enough (at times conflicting) goals
above to justify having multiple versions of KUnit, we want to ensure
that they are meeting their goals, and that we have a process to
ensure that code finds its way into the correct version, that we can
deprecate and remove failed experiments or superseded versions, and
that we can keep pace with upstream kernel releases.

============
The Proposal
============

We propose having two tracks of development: the upstream kernel
(comprising both code that has been upstreamed, and code which is in
the process of being upstreamed -- i.e. is being reviewed on the
mailing lists), and an 'experimental' branch, which contains features
which are yet to be submitted upstream.

Periodically, the 'experimental' branch will be replaced with a new
'experimental' branch based on the latest upstream version. This will
allow us to focus on code based on upstream versions by deprecating
earlier, experimental versions of features. The older experimental
branch will stay around, but receive no new features or support:
customers will need to eventually migrate to either the upstream
kernel (if all of the features they require have been upstreamed), or
to the latest 'experimental' branch (if they still require
non-upstreamed features).

Where practical, 'experimental' branches will try to provide a stable
interface (and/or compatibility layers/shims) such that users can
write tests against an experimental version and expect our changes to
break them infrequently. We don't intend to make any solid guarantee,
however: so users will need to be able to deal with some breakage.

We expect most changes to KUnit to be submitted directly upstream, and
reserve the 'experimental' branch for features where:
a) the design is yet to be proven by early adopters, and is likely to change;
b) the feature is large enough that having a solid staging environment
in which several developers can work is necessary; and/or
c) it depends on existing features only available in the 'experimental' branch.

An example of features suited to the 'experimental' branch are the
mocking systems[2].

=====================
Lifetime of a feature
=====================

A new feature (that for one of the above reasons may require some
experimentation before being sent upstream) would typically be
developed as follows:

1. An initial version of the feature is developed against the latest
'experimental' branch.
2. The feature is tested by users of the 'experimental' branch.
3. It is revised as needed within the 'experimental' branch. If
breaking changes are required, and supporting some form of
compatibility layer is practical, such a layer is implemented to avoid
breaking existing 'experimental' customers.
4. Once the feature is deemed ready, it is submitted to the mailing
lists for the upstream kernel. Any compatibility layers for earlier
experimental version are removed -- this is a 'clean' version of the
change.
5. Development of the change continues on the kernel mailing lists:
any particularly important changes or bugfixes are ported to the
'experimental' branch if feasible, but complete convergence is not
expected at this stage.
6. When the feature is accepted upstream, a new 'experimental' branch
is created based on the new upstream kernel. Development on the old
'experimental' branch ceases.
7. The upstream version of the feature has now superseded the
experimental version. All future development is done based on the
upstream version, either in the new 'experimental' branch, or directly
against the upstream kernel.
8. Any customers still using the experimental version of the feature
will need to update their code to work with the upstream version in
order to update to the new (supported) 'experimental' branch, or to
the upstream kernel.

===========================
What does this mean for me?
===========================

...as a user?
-------------

If you're using the upstream version of KUnit, nothing! Features will
continue to be pushed upstream when they're ready.

If you're using the 'kunit/alpha/master' branch, then you'll need to
switch to the new 'experimental-<version>' branches. Every so often,
we'll create a new 'experimental-<version>' branch, and you'll need to
rebase your code on top of that to keep receiving updated features and
bugfixes. While we intend to make this as painless as possible, there
may be breaking changes which require you to rework some of your
tests.

Once a feature exists in the upstream kernel, the version of that
feature in the 'experimental' branches should stabilise somewhat.

...as a KUnit developer?
------------------------

For most changes/bugfixes which apply cleanly to the upstream version,
you should directly submit these upstream as before. If it is feasible
and useful to backport them to the latest 'experimental' branch (e.g.,
they are bugfixes to code that the experimental branch already has),
please do so.

For more major features (or anything which depends on
yet-to-be-upstreamed code), you should develop them in the latest
'experimental' branch. For the most part, you can treat this the same
way as the old 'kunit/alpha/master' branch, but note that changes will
need to be rebased on top of a newer experimental branch periodically.
It may make sense to use this as an opportunity to tidy the code up
somewhat: by removing compatibility code for deprecated features, or
to merge 'fixup' commits. Once the code has been upstreamed, the next
rebase of the 'experimental' branch should only use the upstream
version: there should not be a separate 'experimental' version of
upstream code save for the development of a new 'experimental' feature
or bugfix.

For more details, see the 'Lifetime of a feature' section above.

=============
In Conclusion
=============

We'll continue to be working with upstream as before, but we intend to
rename and more explicitly specify the purpose of the
'kunit/alpha/master' branch as above. This will require a bit more
work from people using and developing against this 'experimental'
branch, but allows us to continue to use it for features such as the
mocking framework while limiting its divergence from upstream.

Do let us know if this sounds reasonable to you, if it will affect
your use case, or if you have any questions.

Cheers,
-- David & Brendan


[1] https://kunit.googlesource.com/linux/+/refs/heads/kunit/alpha/master
[2] https://kunit.googlesource.com/linux/+/kunit/alpha/master/Documentation/test/usage.rst#187

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4847 bytes --]

Re: kunit: what do we do with the 'kunit/alpha/master' branch?

[Brendan Higgins]

On Tue, Sep 17, 2019 at 11:41 AM David Gow <davidgow@google.com> wrote:
>
> TL;DR: We expect KUnit to be accepted upstream into Linus' branch in
> the next week or two, and we now need to figure out what we are going
> to do with our non-upstream 'kunit/alpha/master' branch.

Given that it has been about a week and we haven't heard any comments,
complaints, or concerns about this. I assume that there are no strong
opinions against this, and people will be generally okay with this
strategy.

As mentioned previously, we are expecting to see KUnit make it into
torvalds/master this merge window (the next week or so), so we will
probably update/rename kunit/alpha/master shortly thereafter.

Cheers

> Hello everyone,
>
> We've put together a rough proposal of what we should do with our
> not-upstream branch, known to most people using it as
> 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline
> appears to be imminent (the KUnit MVP patchset is now in linux-next,
> and the merge window just opened).
>
> ==========
> Background
> ==========
>
> KUnit development is currently split between two versions: the
> 'kunit/alpha/master'[1] git branch, and the version being submitted to
> the upstream Linux kernel. While there are some good reasons to
> continue to have two separate versions of KUnit, at present there is
> some uncertainty around the difference between these versions, and in
> which circumstances each version is useful.
>
> At present, the 'kunit/alpha/master' branch serves a few different
> purposes. It is a place for code not-yet-ready for upstream -- such as
> the mocking framework -- while being developed, while also acting as a
> stable version for customers who do not wish to follow along with the
> changes made during the upstreaming process. Adding to the confusion,
> the name 'kunit/alpha/master' refers to an early (alpha) version of
> KUnit, and the version of KUnit being upstreamed has now diverged
> significantly from this version, requiring significant differences in
> documentation, and requiring a number of changes to tests when porting
> from one version to the other. Finally, it is not clear how the
> 'kunit/alpha/master' version should evolve as features it contains are
> upstreamed.
>
> On the other hand, the version being upstreamed has its own
> complications. It contains significantly fewer features (as features
> such as the mocking frameworks will be upstreamed individually), and
> so is less useful for the average customer. Until each feature is
> upstreamed, it is iterated on rapidly to address comments from the
> kernel community, so in-progress features are not stable enough to
> reasonably build on. Finally, it exists only as a set of patches on
> mailing lists, rather than as a maintained git repository (due to the
> fact that the patches themselves are changing rapidly), making it
> difficult for early adopters to incorporate into their own trees.
>
> Whilst we believe there to be enough (at times conflicting) goals
> above to justify having multiple versions of KUnit, we want to ensure
> that they are meeting their goals, and that we have a process to
> ensure that code finds its way into the correct version, that we can
> deprecate and remove failed experiments or superseded versions, and
> that we can keep pace with upstream kernel releases.
>
> ============
> The Proposal
> ============
>
> We propose having two tracks of development: the upstream kernel
> (comprising both code that has been upstreamed, and code which is in
> the process of being upstreamed -- i.e. is being reviewed on the
> mailing lists), and an 'experimental' branch, which contains features
> which are yet to be submitted upstream.
>
> Periodically, the 'experimental' branch will be replaced with a new
> 'experimental' branch based on the latest upstream version. This will
> allow us to focus on code based on upstream versions by deprecating
> earlier, experimental versions of features. The older experimental
> branch will stay around, but receive no new features or support:
> customers will need to eventually migrate to either the upstream
> kernel (if all of the features they require have been upstreamed), or
> to the latest 'experimental' branch (if they still require
> non-upstreamed features).
>
> Where practical, 'experimental' branches will try to provide a stable
> interface (and/or compatibility layers/shims) such that users can
> write tests against an experimental version and expect our changes to
> break them infrequently. We don't intend to make any solid guarantee,
> however: so users will need to be able to deal with some breakage.
>
> We expect most changes to KUnit to be submitted directly upstream, and
> reserve the 'experimental' branch for features where:
> a) the design is yet to be proven by early adopters, and is likely to change;
> b) the feature is large enough that having a solid staging environment
> in which several developers can work is necessary; and/or
> c) it depends on existing features only available in the 'experimental' branch.
>
> An example of features suited to the 'experimental' branch are the
> mocking systems[2].
>
> =====================
> Lifetime of a feature
> =====================
>
> A new feature (that for one of the above reasons may require some
> experimentation before being sent upstream) would typically be
> developed as follows:
>
> 1. An initial version of the feature is developed against the latest
> 'experimental' branch.
> 2. The feature is tested by users of the 'experimental' branch.
> 3. It is revised as needed within the 'experimental' branch. If
> breaking changes are required, and supporting some form of
> compatibility layer is practical, such a layer is implemented to avoid
> breaking existing 'experimental' customers.
> 4. Once the feature is deemed ready, it is submitted to the mailing
> lists for the upstream kernel. Any compatibility layers for earlier
> experimental version are removed -- this is a 'clean' version of the
> change.
> 5. Development of the change continues on the kernel mailing lists:
> any particularly important changes or bugfixes are ported to the
> 'experimental' branch if feasible, but complete convergence is not
> expected at this stage.
> 6. When the feature is accepted upstream, a new 'experimental' branch
> is created based on the new upstream kernel. Development on the old
> 'experimental' branch ceases.
> 7. The upstream version of the feature has now superseded the
> experimental version. All future development is done based on the
> upstream version, either in the new 'experimental' branch, or directly
> against the upstream kernel.
> 8. Any customers still using the experimental version of the feature
> will need to update their code to work with the upstream version in
> order to update to the new (supported) 'experimental' branch, or to
> the upstream kernel.
>
> ===========================
> What does this mean for me?
> ===========================
>
> ...as a user?
> -------------
>
> If you're using the upstream version of KUnit, nothing! Features will
> continue to be pushed upstream when they're ready.
>
> If you're using the 'kunit/alpha/master' branch, then you'll need to
> switch to the new 'experimental-<version>' branches. Every so often,
> we'll create a new 'experimental-<version>' branch, and you'll need to
> rebase your code on top of that to keep receiving updated features and
> bugfixes. While we intend to make this as painless as possible, there
> may be breaking changes which require you to rework some of your
> tests.
>
> Once a feature exists in the upstream kernel, the version of that
> feature in the 'experimental' branches should stabilise somewhat.
>
> ...as a KUnit developer?
> ------------------------
>
> For most changes/bugfixes which apply cleanly to the upstream version,
> you should directly submit these upstream as before. If it is feasible
> and useful to backport them to the latest 'experimental' branch (e.g.,
> they are bugfixes to code that the experimental branch already has),
> please do so.
>
> For more major features (or anything which depends on
> yet-to-be-upstreamed code), you should develop them in the latest
> 'experimental' branch. For the most part, you can treat this the same
> way as the old 'kunit/alpha/master' branch, but note that changes will
> need to be rebased on top of a newer experimental branch periodically.
> It may make sense to use this as an opportunity to tidy the code up
> somewhat: by removing compatibility code for deprecated features, or
> to merge 'fixup' commits. Once the code has been upstreamed, the next
> rebase of the 'experimental' branch should only use the upstream
> version: there should not be a separate 'experimental' version of
> upstream code save for the development of a new 'experimental' feature
> or bugfix.
>
> For more details, see the 'Lifetime of a feature' section above.
>
> =============
> In Conclusion
> =============
>
> We'll continue to be working with upstream as before, but we intend to
> rename and more explicitly specify the purpose of the
> 'kunit/alpha/master' branch as above. This will require a bit more
> work from people using and developing against this 'experimental'
> branch, but allows us to continue to use it for features such as the
> mocking framework while limiting its divergence from upstream.
>
> Do let us know if this sounds reasonable to you, if it will affect
> your use case, or if you have any questions.
>
> Cheers,
> -- David & Brendan
>
>
> [1] https://kunit.googlesource.com/linux/+/refs/heads/kunit/alpha/master
> [2] https://kunit.googlesource.com/linux/+/kunit/alpha/master/Documentation/test/usage.rst#187

RE: kunit: what do we do with the 'kunit/alpha/master' branch?

[Siebert, David]

Right now I am focused on the mocking and getting that to work. I am still have some very strange issues with mocking some function calls including kernel panics.
Your plan seems logical but just how useful is it without mocking?
-----Original Message-----
From: Brendan Higgins [mailto:brendanhiggins@google.com] 
Sent: Monday, September 23, 2019 5:42 PM
To: David Gow <davidgow@google.com>
Cc: David Chiang <davidchiang@google.com>; Siebert, David (US) <David.Siebert@L3Harris.com>; Kees Cook <keescook@chromium.org>; Mike Salvatore <mike.salvatore@canonical.com>; Pei Huang <peihuang@google.com>; Sagi Shahar <sagis@google.com>; Sangsu Ha <sangsu.ha@samsung.com>; Shuah Khan <shuah@kernel.org>; kunit-dev@googlegroups.com; open list:KERNEL SELFTEST FRAMEWORK <linux-kselftest@vger.kernel.org>
Subject: Re: kunit: what do we do with the 'kunit/alpha/master' branch?

On Tue, Sep 17, 2019 at 11:41 AM David Gow <davidgow@google.com> wrote:
>
> TL;DR: We expect KUnit to be accepted upstream into Linus' branch in 
> the next week or two, and we now need to figure out what we are going 
> to do with our non-upstream 'kunit/alpha/master' branch.

Given that it has been about a week and we haven't heard any comments, complaints, or concerns about this. I assume that there are no strong opinions against this, and people will be generally okay with this strategy.

As mentioned previously, we are expecting to see KUnit make it into torvalds/master this merge window (the next week or so), so we will probably update/rename kunit/alpha/master shortly thereafter.

Cheers

> Hello everyone,
>
> We've put together a rough proposal of what we should do with our 
> not-upstream branch, known to most people using it as 
> 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline 
> appears to be imminent (the KUnit MVP patchset is now in linux-next, 
> and the merge window just opened).
>
> ==========
> Background
> ==========
>
> KUnit development is currently split between two versions: the 
> 'kunit/alpha/master'[1] git branch, and the version being submitted to 
> the upstream Linux kernel. While there are some good reasons to 
> continue to have two separate versions of KUnit, at present there is 
> some uncertainty around the difference between these versions, and in 
> which circumstances each version is useful.
>
> At present, the 'kunit/alpha/master' branch serves a few different 
> purposes. It is a place for code not-yet-ready for upstream -- such as 
> the mocking framework -- while being developed, while also acting as a 
> stable version for customers who do not wish to follow along with the 
> changes made during the upstreaming process. Adding to the confusion, 
> the name 'kunit/alpha/master' refers to an early (alpha) version of 
> KUnit, and the version of KUnit being upstreamed has now diverged 
> significantly from this version, requiring significant differences in 
> documentation, and requiring a number of changes to tests when porting 
> from one version to the other. Finally, it is not clear how the 
> 'kunit/alpha/master' version should evolve as features it contains are 
> upstreamed.
>
> On the other hand, the version being upstreamed has its own 
> complications. It contains significantly fewer features (as features 
> such as the mocking frameworks will be upstreamed individually), and 
> so is less useful for the average customer. Until each feature is 
> upstreamed, it is iterated on rapidly to address comments from the 
> kernel community, so in-progress features are not stable enough to 
> reasonably build on. Finally, it exists only as a set of patches on 
> mailing lists, rather than as a maintained git repository (due to the 
> fact that the patches themselves are changing rapidly), making it 
> difficult for early adopters to incorporate into their own trees.
>
> Whilst we believe there to be enough (at times conflicting) goals 
> above to justify having multiple versions of KUnit, we want to ensure 
> that they are meeting their goals, and that we have a process to 
> ensure that code finds its way into the correct version, that we can 
> deprecate and remove failed experiments or superseded versions, and 
> that we can keep pace with upstream kernel releases.
>
> ============
> The Proposal
> ============
>
> We propose having two tracks of development: the upstream kernel 
> (comprising both code that has been upstreamed, and code which is in 
> the process of being upstreamed -- i.e. is being reviewed on the 
> mailing lists), and an 'experimental' branch, which contains features 
> which are yet to be submitted upstream.
>
> Periodically, the 'experimental' branch will be replaced with a new 
> 'experimental' branch based on the latest upstream version. This will 
> allow us to focus on code based on upstream versions by deprecating 
> earlier, experimental versions of features. The older experimental 
> branch will stay around, but receive no new features or support:
> customers will need to eventually migrate to either the upstream 
> kernel (if all of the features they require have been upstreamed), or 
> to the latest 'experimental' branch (if they still require 
> non-upstreamed features).
>
> Where practical, 'experimental' branches will try to provide a stable 
> interface (and/or compatibility layers/shims) such that users can 
> write tests against an experimental version and expect our changes to 
> break them infrequently. We don't intend to make any solid guarantee,
> however: so users will need to be able to deal with some breakage.
>
> We expect most changes to KUnit to be submitted directly upstream, and 
> reserve the 'experimental' branch for features where:
> a) the design is yet to be proven by early adopters, and is likely to 
> change;
> b) the feature is large enough that having a solid staging environment 
> in which several developers can work is necessary; and/or
> c) it depends on existing features only available in the 'experimental' branch.
>
> An example of features suited to the 'experimental' branch are the 
> mocking systems[2].
>
> =====================
> Lifetime of a feature
> =====================
>
> A new feature (that for one of the above reasons may require some 
> experimentation before being sent upstream) would typically be 
> developed as follows:
>
> 1. An initial version of the feature is developed against the latest 
> 'experimental' branch.
> 2. The feature is tested by users of the 'experimental' branch.
> 3. It is revised as needed within the 'experimental' branch. If 
> breaking changes are required, and supporting some form of 
> compatibility layer is practical, such a layer is implemented to avoid 
> breaking existing 'experimental' customers.
> 4. Once the feature is deemed ready, it is submitted to the mailing 
> lists for the upstream kernel. Any compatibility layers for earlier 
> experimental version are removed -- this is a 'clean' version of the 
> change.
> 5. Development of the change continues on the kernel mailing lists:
> any particularly important changes or bugfixes are ported to the 
> 'experimental' branch if feasible, but complete convergence is not 
> expected at this stage.
> 6. When the feature is accepted upstream, a new 'experimental' branch 
> is created based on the new upstream kernel. Development on the old 
> 'experimental' branch ceases.
> 7. The upstream version of the feature has now superseded the 
> experimental version. All future development is done based on the 
> upstream version, either in the new 'experimental' branch, or directly 
> against the upstream kernel.
> 8. Any customers still using the experimental version of the feature 
> will need to update their code to work with the upstream version in 
> order to update to the new (supported) 'experimental' branch, or to 
> the upstream kernel.
>
> ===========================
> What does this mean for me?
> ===========================
>
> ...as a user?
> -------------
>
> If you're using the upstream version of KUnit, nothing! Features will 
> continue to be pushed upstream when they're ready.
>
> If you're using the 'kunit/alpha/master' branch, then you'll need to 
> switch to the new 'experimental-<version>' branches. Every so often, 
> we'll create a new 'experimental-<version>' branch, and you'll need to 
> rebase your code on top of that to keep receiving updated features and 
> bugfixes. While we intend to make this as painless as possible, there 
> may be breaking changes which require you to rework some of your 
> tests.
>
> Once a feature exists in the upstream kernel, the version of that 
> feature in the 'experimental' branches should stabilise somewhat.
>
> ...as a KUnit developer?
> ------------------------
>
> For most changes/bugfixes which apply cleanly to the upstream version, 
> you should directly submit these upstream as before. If it is feasible 
> and useful to backport them to the latest 'experimental' branch (e.g., 
> they are bugfixes to code that the experimental branch already has), 
> please do so.
>
> For more major features (or anything which depends on 
> yet-to-be-upstreamed code), you should develop them in the latest 
> 'experimental' branch. For the most part, you can treat this the same 
> way as the old 'kunit/alpha/master' branch, but note that changes will 
> need to be rebased on top of a newer experimental branch periodically.
> It may make sense to use this as an opportunity to tidy the code up
> somewhat: by removing compatibility code for deprecated features, or 
> to merge 'fixup' commits. Once the code has been upstreamed, the next 
> rebase of the 'experimental' branch should only use the upstream
> version: there should not be a separate 'experimental' version of 
> upstream code save for the development of a new 'experimental' feature 
> or bugfix.
>
> For more details, see the 'Lifetime of a feature' section above.
>
> =============
> In Conclusion
> =============
>
> We'll continue to be working with upstream as before, but we intend to 
> rename and more explicitly specify the purpose of the 
> 'kunit/alpha/master' branch as above. This will require a bit more 
> work from people using and developing against this 'experimental'
> branch, but allows us to continue to use it for features such as the 
> mocking framework while limiting its divergence from upstream.
>
> Do let us know if this sounds reasonable to you, if it will affect 
> your use case, or if you have any questions.
>
> Cheers,
> -- David & Brendan
>
>
> [1] 
> https://kunit.googlesource.com/linux/+/refs/heads/kunit/alpha/master
> [2] 
> https://kunit.googlesource.com/linux/+/kunit/alpha/master/Documentatio
> n/test/usage.rst#187

  

CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient and may contain material that is proprietary, confidential, privileged or otherwise legally protected or restricted under applicable government laws. Any review, disclosure, distributing or other use without expressed permission of the sender is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies without reading, printing, or saving.

Re: kunit: what do we do with the 'kunit/alpha/master' branch?

[shuah]

On 9/23/19 3:41 PM, Brendan Higgins wrote:
> On Tue, Sep 17, 2019 at 11:41 AM David Gow <davidgow@google.com> wrote:
>>
>> TL;DR: We expect KUnit to be accepted upstream into Linus' branch in
>> the next week or two, and we now need to figure out what we are going
>> to do with our non-upstream 'kunit/alpha/master' branch.
> 
> Given that it has been about a week and we haven't heard any comments,
> complaints, or concerns about this. I assume that there are no strong
> opinions against this, and people will be generally okay with this
> strategy.
> 
> As mentioned previously, we are expecting to see KUnit make it into
> torvalds/master this merge window (the next week or so), so we will
> probably update/rename kunit/alpha/master shortly thereafter.
> 
> Cheers
> 
>> Hello everyone,
>>
>> We've put together a rough proposal of what we should do with our
>> not-upstream branch, known to most people using it as
>> 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline
>> appears to be imminent (the KUnit MVP patchset is now in linux-next,
>> and the merge window just opened).
>>
>> ==========
>> Background
>> ==========
>>
>> KUnit development is currently split between two versions: the
>> 'kunit/alpha/master'[1] git branch, and the version being submitted to
>> the upstream Linux kernel. While there are some good reasons to
>> continue to have two separate versions of KUnit, at present there is
>> some uncertainty around the difference between these versions, and in
>> which circumstances each version is useful.
>>
>> At present, the 'kunit/alpha/master' branch serves a few different
>> purposes. It is a place for code not-yet-ready for upstream -- such as
>> the mocking framework -- while being developed, while also acting as a
>> stable version for customers who do not wish to follow along with the
>> changes made during the upstreaming process. Adding to the confusion,
>> the name 'kunit/alpha/master' refers to an early (alpha) version of
>> KUnit, and the version of KUnit being upstreamed has now diverged
>> significantly from this version, requiring significant differences in
>> documentation, and requiring a number of changes to tests when porting
>> from one version to the other. Finally, it is not clear how the
>> 'kunit/alpha/master' version should evolve as features it contains are
>> upstreamed.
>>
>> On the other hand, the version being upstreamed has its own
>> complications. It contains significantly fewer features (as features
>> such as the mocking frameworks will be upstreamed individually), and
>> so is less useful for the average customer. Until each feature is
>> upstreamed, it is iterated on rapidly to address comments from the
>> kernel community, so in-progress features are not stable enough to
>> reasonably build on. Finally, it exists only as a set of patches on
>> mailing lists, rather than as a maintained git repository (due to the
>> fact that the patches themselves are changing rapidly), making it
>> difficult for early adopters to incorporate into their own trees.
>>
>> Whilst we believe there to be enough (at times conflicting) goals
>> above to justify having multiple versions of KUnit, we want to ensure
>> that they are meeting their goals, and that we have a process to
>> ensure that code finds its way into the correct version, that we can
>> deprecate and remove failed experiments or superseded versions, and
>> that we can keep pace with upstream kernel releases.
>>
>> ============
>> The Proposal
>> ============
>>
>> We propose having two tracks of development: the upstream kernel
>> (comprising both code that has been upstreamed, and code which is in
>> the process of being upstreamed -- i.e. is being reviewed on the
>> mailing lists), and an 'experimental' branch, which contains features
>> which are yet to be submitted upstream.
>>

My concern with this approach is either one could outdated. is there a
reason continue in parallel mode. I would rathet see development happen
upstream so we don't have lot of code to be upstreamed sitting in an
experimental branch while upstream keeps moving. It is given that they
will diverge.

thanks,
-- Shuah

Re: kunit: what do we do with the 'kunit/alpha/master' branch?

[Brendan Higgins]

On Mon, Sep 23, 2019 at 2:45 PM Siebert, David
<David.Siebert@l3harris.com> wrote:
>
>
> Right now I am focused on the mocking and getting that to work. I am still have some very strange issues with mocking some function calls including kernel panics.

Okay, I just responded to your last email on the topic.

> Your plan seems logical but just how useful is it without mocking?

It depends on what you are trying to test. We have a number of tests
so far that do not require mocking (there are several in the initial
upstream patchset itself). A lot more can be tested with the
appropriate use of dependency injection.

In anycase, we are still planning on upstreaming support for mocking,
it is just going to take some time.

Re: kunit: what do we do with the 'kunit/alpha/master' branch?

[David Gow]

On Mon, Sep 23, 2019 at 2:52 PM shuah <shuah@kernel.org> wrote:
>
> My concern with this approach is either one could outdated. is there a
> reason continue in parallel mode. I would rathet see development happen
> upstream so we don't have lot of code to be upstreamed sitting in an
> experimental branch while upstream keeps moving. It is given that they
> will diverge.

I definitely appreciate that, and the aim certainly is to have most
changes go straight upstream without passing through the
'experimental' branch first.

The real purpose of the 'experimental' branch is to have somewhere to
keep the mocking functionality before we're ready to upstream it.
Given that there are already people using the current version of the
mocking framework, we want to provide a smooth-ish path to upstream by
providing this branch which is at least closer to upstream than when
we are now (and that'll stay as close to upstream as possible through
regular rebasing, rather than staying 'stuck' on the older versions).

Ultimately, the 'experimental' branch should go away once all of the
pieces of the mocking framework have made it upstream, so this is
really outlining a transition plan. Now, if we thought that the
mocking changes were sufficiently ready, we could try to upstream them
soon and kill the old 'kunit/alpha/master' branch without having the
intermediate 'experimental' branch. As it is, though, I don't think we
feel the mocking framework will be ready for quite some time.

So, yes, they'll diverge a bit, but hopefully a bit less than with
what we're going now with our 'kunit/master/alpha' branch. Most
development that doesn't relate to the mocking system should go
directly upstream, and we'll try to minimise the divergence in the
'experimental' branch by rebasing it regularly on top of any upstream
changes.

When in doubt, though, KUnit changes should definitely be going
upstream rather than to this 'experimental' branch. Hopefully that
minimises the divergence and makes this more tolerable.

Cheers,
-- David

Re: kunit: what do we do with the 'kunit/alpha/master' branch?

[shuah]

On 9/23/19 5:00 PM, David Gow wrote:
> On Mon, Sep 23, 2019 at 2:52 PM shuah <shuah@kernel.org> wrote:
>>
>> My concern with this approach is either one could outdated. is there a
>> reason continue in parallel mode. I would rathet see development happen
>> upstream so we don't have lot of code to be upstreamed sitting in an
>> experimental branch while upstream keeps moving. It is given that they
>> will diverge.
> 
> I definitely appreciate that, and the aim certainly is to have most
> changes go straight upstream without passing through the
> 'experimental' branch first.
> 
> The real purpose of the 'experimental' branch is to have somewhere to
> keep the mocking functionality before we're ready to upstream it.
> Given that there are already people using the current version of the
> mocking framework, we want to provide a smooth-ish path to upstream by
> providing this branch which is at least closer to upstream than when
> we are now (and that'll stay as close to upstream as possible through
> regular rebasing, rather than staying 'stuck' on the older versions).
> 

What I would like to see is a freeze on the experimental branch as soon
as KUnit goes into mainline (which is really at the end of this week)

Start draining the experimental branch with a goal to get all everything 
currently staged there mainlined.

Please define clear sunset date for the experimental branch. Without
that we are looking at a lot of pain in the future.

thanks,
-- Shuah

Re: kunit: what do we do with the 'kunit/alpha/master' branch?

[Brendan Higgins]

On Mon, Sep 23, 2019 at 05:19:58PM -0600, shuah wrote:
> On 9/23/19 5:00 PM, David Gow wrote:
> > On Mon, Sep 23, 2019 at 2:52 PM shuah <shuah@kernel.org> wrote:
> > > 
> > > My concern with this approach is either one could outdated. is there a
> > > reason continue in parallel mode. I would rathet see development happen
> > > upstream so we don't have lot of code to be upstreamed sitting in an
> > > experimental branch while upstream keeps moving. It is given that they
> > > will diverge.
> > 
> > I definitely appreciate that, and the aim certainly is to have most
> > changes go straight upstream without passing through the
> > 'experimental' branch first.
> > 
> > The real purpose of the 'experimental' branch is to have somewhere to
> > keep the mocking functionality before we're ready to upstream it.
> > Given that there are already people using the current version of the
> > mocking framework, we want to provide a smooth-ish path to upstream by
> > providing this branch which is at least closer to upstream than when
> > we are now (and that'll stay as close to upstream as possible through
> > regular rebasing, rather than staying 'stuck' on the older versions).
> > 
> 
> What I would like to see is a freeze on the experimental branch as soon
> as KUnit goes into mainline (which is really at the end of this week)
> 
> Start draining the experimental branch with a goal to get all everything
> currently staged there mainlined.

Yep, that's the plan. Sorry, if that wasn't clear, but we were trying to
be intentionally vague about some things to give ourselves room to
maneuver. In anycase, we actually want to be pretty aggressive dropping
things from experimental as soon as the feature makes it upstream.

> Please define clear sunset date for the experimental branch. Without
> that we are looking at a lot of pain in the future.

I would like to, but I find being able to predict how long it takes to
do something upstream to be pretty hard. Especially with large features
where people are likely to have strong opinions on.

To give you and idea for upstreaming mocking stuff, I see 3 major
components:
 - Basic "class" mocking and parameter matchers (might be able to split
   them into two parts) - about 7 patches.
 - Arbitrary function mocking and spying - currently just a couple
   patches, but I expect a lot of discussion. I am actually hoping we
   can use some of Knut's work for this. So this is probably a pretty
   big project.
 - Platform/hardware mocking and faking - currently also just a handful
   of patches, but I also expect to get a lot of discussion on this.

I could easily see all this taking well over a year; nevertheless, we
want to work on other things as well. In fact, from my talk at LPC, it
seems like the general consensus is that the mocking stuff is not a very
high priority in terms of what the people there wanted to see.

So I definitely want to see this all go upstream as soon as possible,
nothing would make me happier; however, I think the reality is that
there is too much uncertainty to paint a hard deadline.

I think it probably makes sense to try to set a roadmap, but I think it
will probably end up being pretty rough past Q4.

Nevertheless, I am open to alternatives. I know that trying to maintain
a staging repo is a lot of work with no long term benefit, and I think
any amount of worked saved there can be spent on more useful things.
Part of the reason we shared this publicly was that we hoped that
smarter more experienced people might be able to save us from some of
this pain :-)

Looking forward to hearing people's thoughts!