* kunit: what do we do with the 'kunit/alpha/master' branch? @ 2019-09-17 18:41 David Gow 2019-09-23 21:41 ` Brendan Higgins 0 siblings, 1 reply; 8+ messages in thread From: David Gow @ 2019-09-17 18:41 UTC (permalink / raw) To: Brendan Higgins, David Chiang, David Siebert, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, Shuah Khan Cc: kunit-dev, linux-kselftest [-- Attachment #1: Type: text/plain, Size: 8886 bytes --] TL;DR: We expect KUnit to be accepted upstream into Linus' branch in the next week or two, and we now need to figure out what we are going to do with our non-upstream 'kunit/alpha/master' branch. Hello everyone, We've put together a rough proposal of what we should do with our not-upstream branch, known to most people using it as 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline appears to be imminent (the KUnit MVP patchset is now in linux-next, and the merge window just opened). ========== Background ========== KUnit development is currently split between two versions: the 'kunit/alpha/master'[1] git branch, and the version being submitted to the upstream Linux kernel. While there are some good reasons to continue to have two separate versions of KUnit, at present there is some uncertainty around the difference between these versions, and in which circumstances each version is useful. At present, the 'kunit/alpha/master' branch serves a few different purposes. It is a place for code not-yet-ready for upstream -- such as the mocking framework -- while being developed, while also acting as a stable version for customers who do not wish to follow along with the changes made during the upstreaming process. Adding to the confusion, the name 'kunit/alpha/master' refers to an early (alpha) version of KUnit, and the version of KUnit being upstreamed has now diverged significantly from this version, requiring significant differences in documentation, and requiring a number of changes to tests when porting from one version to the other. Finally, it is not clear how the 'kunit/alpha/master' version should evolve as features it contains are upstreamed. On the other hand, the version being upstreamed has its own complications. It contains significantly fewer features (as features such as the mocking frameworks will be upstreamed individually), and so is less useful for the average customer. Until each feature is upstreamed, it is iterated on rapidly to address comments from the kernel community, so in-progress features are not stable enough to reasonably build on. Finally, it exists only as a set of patches on mailing lists, rather than as a maintained git repository (due to the fact that the patches themselves are changing rapidly), making it difficult for early adopters to incorporate into their own trees. Whilst we believe there to be enough (at times conflicting) goals above to justify having multiple versions of KUnit, we want to ensure that they are meeting their goals, and that we have a process to ensure that code finds its way into the correct version, that we can deprecate and remove failed experiments or superseded versions, and that we can keep pace with upstream kernel releases. ============ The Proposal ============ We propose having two tracks of development: the upstream kernel (comprising both code that has been upstreamed, and code which is in the process of being upstreamed -- i.e. is being reviewed on the mailing lists), and an 'experimental' branch, which contains features which are yet to be submitted upstream. Periodically, the 'experimental' branch will be replaced with a new 'experimental' branch based on the latest upstream version. This will allow us to focus on code based on upstream versions by deprecating earlier, experimental versions of features. The older experimental branch will stay around, but receive no new features or support: customers will need to eventually migrate to either the upstream kernel (if all of the features they require have been upstreamed), or to the latest 'experimental' branch (if they still require non-upstreamed features). Where practical, 'experimental' branches will try to provide a stable interface (and/or compatibility layers/shims) such that users can write tests against an experimental version and expect our changes to break them infrequently. We don't intend to make any solid guarantee, however: so users will need to be able to deal with some breakage. We expect most changes to KUnit to be submitted directly upstream, and reserve the 'experimental' branch for features where: a) the design is yet to be proven by early adopters, and is likely to change; b) the feature is large enough that having a solid staging environment in which several developers can work is necessary; and/or c) it depends on existing features only available in the 'experimental' branch. An example of features suited to the 'experimental' branch are the mocking systems[2]. ===================== Lifetime of a feature ===================== A new feature (that for one of the above reasons may require some experimentation before being sent upstream) would typically be developed as follows: 1. An initial version of the feature is developed against the latest 'experimental' branch. 2. The feature is tested by users of the 'experimental' branch. 3. It is revised as needed within the 'experimental' branch. If breaking changes are required, and supporting some form of compatibility layer is practical, such a layer is implemented to avoid breaking existing 'experimental' customers. 4. Once the feature is deemed ready, it is submitted to the mailing lists for the upstream kernel. Any compatibility layers for earlier experimental version are removed -- this is a 'clean' version of the change. 5. Development of the change continues on the kernel mailing lists: any particularly important changes or bugfixes are ported to the 'experimental' branch if feasible, but complete convergence is not expected at this stage. 6. When the feature is accepted upstream, a new 'experimental' branch is created based on the new upstream kernel. Development on the old 'experimental' branch ceases. 7. The upstream version of the feature has now superseded the experimental version. All future development is done based on the upstream version, either in the new 'experimental' branch, or directly against the upstream kernel. 8. Any customers still using the experimental version of the feature will need to update their code to work with the upstream version in order to update to the new (supported) 'experimental' branch, or to the upstream kernel. =========================== What does this mean for me? =========================== ...as a user? ------------- If you're using the upstream version of KUnit, nothing! Features will continue to be pushed upstream when they're ready. If you're using the 'kunit/alpha/master' branch, then you'll need to switch to the new 'experimental-<version>' branches. Every so often, we'll create a new 'experimental-<version>' branch, and you'll need to rebase your code on top of that to keep receiving updated features and bugfixes. While we intend to make this as painless as possible, there may be breaking changes which require you to rework some of your tests. Once a feature exists in the upstream kernel, the version of that feature in the 'experimental' branches should stabilise somewhat. ...as a KUnit developer? ------------------------ For most changes/bugfixes which apply cleanly to the upstream version, you should directly submit these upstream as before. If it is feasible and useful to backport them to the latest 'experimental' branch (e.g., they are bugfixes to code that the experimental branch already has), please do so. For more major features (or anything which depends on yet-to-be-upstreamed code), you should develop them in the latest 'experimental' branch. For the most part, you can treat this the same way as the old 'kunit/alpha/master' branch, but note that changes will need to be rebased on top of a newer experimental branch periodically. It may make sense to use this as an opportunity to tidy the code up somewhat: by removing compatibility code for deprecated features, or to merge 'fixup' commits. Once the code has been upstreamed, the next rebase of the 'experimental' branch should only use the upstream version: there should not be a separate 'experimental' version of upstream code save for the development of a new 'experimental' feature or bugfix. For more details, see the 'Lifetime of a feature' section above. ============= In Conclusion ============= We'll continue to be working with upstream as before, but we intend to rename and more explicitly specify the purpose of the 'kunit/alpha/master' branch as above. This will require a bit more work from people using and developing against this 'experimental' branch, but allows us to continue to use it for features such as the mocking framework while limiting its divergence from upstream. Do let us know if this sounds reasonable to you, if it will affect your use case, or if you have any questions. Cheers, -- David & Brendan [1] https://kunit.googlesource.com/linux/+/refs/heads/kunit/alpha/master [2] https://kunit.googlesource.com/linux/+/kunit/alpha/master/Documentation/test/usage.rst#187 [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4847 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-17 18:41 kunit: what do we do with the 'kunit/alpha/master' branch? David Gow @ 2019-09-23 21:41 ` Brendan Higgins 2019-09-23 21:45 ` Siebert, David 2019-09-23 21:52 ` shuah 0 siblings, 2 replies; 8+ messages in thread From: Brendan Higgins @ 2019-09-23 21:41 UTC (permalink / raw) To: David Gow Cc: David Chiang, David Siebert, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, Shuah Khan, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK On Tue, Sep 17, 2019 at 11:41 AM David Gow <davidgow@google.com> wrote: > > TL;DR: We expect KUnit to be accepted upstream into Linus' branch in > the next week or two, and we now need to figure out what we are going > to do with our non-upstream 'kunit/alpha/master' branch. Given that it has been about a week and we haven't heard any comments, complaints, or concerns about this. I assume that there are no strong opinions against this, and people will be generally okay with this strategy. As mentioned previously, we are expecting to see KUnit make it into torvalds/master this merge window (the next week or so), so we will probably update/rename kunit/alpha/master shortly thereafter. Cheers > Hello everyone, > > We've put together a rough proposal of what we should do with our > not-upstream branch, known to most people using it as > 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline > appears to be imminent (the KUnit MVP patchset is now in linux-next, > and the merge window just opened). > > ========== > Background > ========== > > KUnit development is currently split between two versions: the > 'kunit/alpha/master'[1] git branch, and the version being submitted to > the upstream Linux kernel. While there are some good reasons to > continue to have two separate versions of KUnit, at present there is > some uncertainty around the difference between these versions, and in > which circumstances each version is useful. > > At present, the 'kunit/alpha/master' branch serves a few different > purposes. It is a place for code not-yet-ready for upstream -- such as > the mocking framework -- while being developed, while also acting as a > stable version for customers who do not wish to follow along with the > changes made during the upstreaming process. Adding to the confusion, > the name 'kunit/alpha/master' refers to an early (alpha) version of > KUnit, and the version of KUnit being upstreamed has now diverged > significantly from this version, requiring significant differences in > documentation, and requiring a number of changes to tests when porting > from one version to the other. Finally, it is not clear how the > 'kunit/alpha/master' version should evolve as features it contains are > upstreamed. > > On the other hand, the version being upstreamed has its own > complications. It contains significantly fewer features (as features > such as the mocking frameworks will be upstreamed individually), and > so is less useful for the average customer. Until each feature is > upstreamed, it is iterated on rapidly to address comments from the > kernel community, so in-progress features are not stable enough to > reasonably build on. Finally, it exists only as a set of patches on > mailing lists, rather than as a maintained git repository (due to the > fact that the patches themselves are changing rapidly), making it > difficult for early adopters to incorporate into their own trees. > > Whilst we believe there to be enough (at times conflicting) goals > above to justify having multiple versions of KUnit, we want to ensure > that they are meeting their goals, and that we have a process to > ensure that code finds its way into the correct version, that we can > deprecate and remove failed experiments or superseded versions, and > that we can keep pace with upstream kernel releases. > > ============ > The Proposal > ============ > > We propose having two tracks of development: the upstream kernel > (comprising both code that has been upstreamed, and code which is in > the process of being upstreamed -- i.e. is being reviewed on the > mailing lists), and an 'experimental' branch, which contains features > which are yet to be submitted upstream. > > Periodically, the 'experimental' branch will be replaced with a new > 'experimental' branch based on the latest upstream version. This will > allow us to focus on code based on upstream versions by deprecating > earlier, experimental versions of features. The older experimental > branch will stay around, but receive no new features or support: > customers will need to eventually migrate to either the upstream > kernel (if all of the features they require have been upstreamed), or > to the latest 'experimental' branch (if they still require > non-upstreamed features). > > Where practical, 'experimental' branches will try to provide a stable > interface (and/or compatibility layers/shims) such that users can > write tests against an experimental version and expect our changes to > break them infrequently. We don't intend to make any solid guarantee, > however: so users will need to be able to deal with some breakage. > > We expect most changes to KUnit to be submitted directly upstream, and > reserve the 'experimental' branch for features where: > a) the design is yet to be proven by early adopters, and is likely to change; > b) the feature is large enough that having a solid staging environment > in which several developers can work is necessary; and/or > c) it depends on existing features only available in the 'experimental' branch. > > An example of features suited to the 'experimental' branch are the > mocking systems[2]. > > ===================== > Lifetime of a feature > ===================== > > A new feature (that for one of the above reasons may require some > experimentation before being sent upstream) would typically be > developed as follows: > > 1. An initial version of the feature is developed against the latest > 'experimental' branch. > 2. The feature is tested by users of the 'experimental' branch. > 3. It is revised as needed within the 'experimental' branch. If > breaking changes are required, and supporting some form of > compatibility layer is practical, such a layer is implemented to avoid > breaking existing 'experimental' customers. > 4. Once the feature is deemed ready, it is submitted to the mailing > lists for the upstream kernel. Any compatibility layers for earlier > experimental version are removed -- this is a 'clean' version of the > change. > 5. Development of the change continues on the kernel mailing lists: > any particularly important changes or bugfixes are ported to the > 'experimental' branch if feasible, but complete convergence is not > expected at this stage. > 6. When the feature is accepted upstream, a new 'experimental' branch > is created based on the new upstream kernel. Development on the old > 'experimental' branch ceases. > 7. The upstream version of the feature has now superseded the > experimental version. All future development is done based on the > upstream version, either in the new 'experimental' branch, or directly > against the upstream kernel. > 8. Any customers still using the experimental version of the feature > will need to update their code to work with the upstream version in > order to update to the new (supported) 'experimental' branch, or to > the upstream kernel. > > =========================== > What does this mean for me? > =========================== > > ...as a user? > ------------- > > If you're using the upstream version of KUnit, nothing! Features will > continue to be pushed upstream when they're ready. > > If you're using the 'kunit/alpha/master' branch, then you'll need to > switch to the new 'experimental-<version>' branches. Every so often, > we'll create a new 'experimental-<version>' branch, and you'll need to > rebase your code on top of that to keep receiving updated features and > bugfixes. While we intend to make this as painless as possible, there > may be breaking changes which require you to rework some of your > tests. > > Once a feature exists in the upstream kernel, the version of that > feature in the 'experimental' branches should stabilise somewhat. > > ...as a KUnit developer? > ------------------------ > > For most changes/bugfixes which apply cleanly to the upstream version, > you should directly submit these upstream as before. If it is feasible > and useful to backport them to the latest 'experimental' branch (e.g., > they are bugfixes to code that the experimental branch already has), > please do so. > > For more major features (or anything which depends on > yet-to-be-upstreamed code), you should develop them in the latest > 'experimental' branch. For the most part, you can treat this the same > way as the old 'kunit/alpha/master' branch, but note that changes will > need to be rebased on top of a newer experimental branch periodically. > It may make sense to use this as an opportunity to tidy the code up > somewhat: by removing compatibility code for deprecated features, or > to merge 'fixup' commits. Once the code has been upstreamed, the next > rebase of the 'experimental' branch should only use the upstream > version: there should not be a separate 'experimental' version of > upstream code save for the development of a new 'experimental' feature > or bugfix. > > For more details, see the 'Lifetime of a feature' section above. > > ============= > In Conclusion > ============= > > We'll continue to be working with upstream as before, but we intend to > rename and more explicitly specify the purpose of the > 'kunit/alpha/master' branch as above. This will require a bit more > work from people using and developing against this 'experimental' > branch, but allows us to continue to use it for features such as the > mocking framework while limiting its divergence from upstream. > > Do let us know if this sounds reasonable to you, if it will affect > your use case, or if you have any questions. > > Cheers, > -- David & Brendan > > > [1] https://kunit.googlesource.com/linux/+/refs/heads/kunit/alpha/master > [2] https://kunit.googlesource.com/linux/+/kunit/alpha/master/Documentation/test/usage.rst#187 ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-23 21:41 ` Brendan Higgins @ 2019-09-23 21:45 ` Siebert, David 2019-09-23 22:26 ` Brendan Higgins 2019-09-23 21:52 ` shuah 1 sibling, 1 reply; 8+ messages in thread From: Siebert, David @ 2019-09-23 21:45 UTC (permalink / raw) To: Brendan Higgins, David Gow Cc: David Chiang, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, Shuah Khan, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK Right now I am focused on the mocking and getting that to work. I am still have some very strange issues with mocking some function calls including kernel panics. Your plan seems logical but just how useful is it without mocking? -----Original Message----- From: Brendan Higgins [mailto:brendanhiggins@google.com] Sent: Monday, September 23, 2019 5:42 PM To: David Gow <davidgow@google.com> Cc: David Chiang <davidchiang@google.com>; Siebert, David (US) <David.Siebert@L3Harris.com>; Kees Cook <keescook@chromium.org>; Mike Salvatore <mike.salvatore@canonical.com>; Pei Huang <peihuang@google.com>; Sagi Shahar <sagis@google.com>; Sangsu Ha <sangsu.ha@samsung.com>; Shuah Khan <shuah@kernel.org>; kunit-dev@googlegroups.com; open list:KERNEL SELFTEST FRAMEWORK <linux-kselftest@vger.kernel.org> Subject: Re: kunit: what do we do with the 'kunit/alpha/master' branch? On Tue, Sep 17, 2019 at 11:41 AM David Gow <davidgow@google.com> wrote: > > TL;DR: We expect KUnit to be accepted upstream into Linus' branch in > the next week or two, and we now need to figure out what we are going > to do with our non-upstream 'kunit/alpha/master' branch. Given that it has been about a week and we haven't heard any comments, complaints, or concerns about this. I assume that there are no strong opinions against this, and people will be generally okay with this strategy. As mentioned previously, we are expecting to see KUnit make it into torvalds/master this merge window (the next week or so), so we will probably update/rename kunit/alpha/master shortly thereafter. Cheers > Hello everyone, > > We've put together a rough proposal of what we should do with our > not-upstream branch, known to most people using it as > 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline > appears to be imminent (the KUnit MVP patchset is now in linux-next, > and the merge window just opened). > > ========== > Background > ========== > > KUnit development is currently split between two versions: the > 'kunit/alpha/master'[1] git branch, and the version being submitted to > the upstream Linux kernel. While there are some good reasons to > continue to have two separate versions of KUnit, at present there is > some uncertainty around the difference between these versions, and in > which circumstances each version is useful. > > At present, the 'kunit/alpha/master' branch serves a few different > purposes. It is a place for code not-yet-ready for upstream -- such as > the mocking framework -- while being developed, while also acting as a > stable version for customers who do not wish to follow along with the > changes made during the upstreaming process. Adding to the confusion, > the name 'kunit/alpha/master' refers to an early (alpha) version of > KUnit, and the version of KUnit being upstreamed has now diverged > significantly from this version, requiring significant differences in > documentation, and requiring a number of changes to tests when porting > from one version to the other. Finally, it is not clear how the > 'kunit/alpha/master' version should evolve as features it contains are > upstreamed. > > On the other hand, the version being upstreamed has its own > complications. It contains significantly fewer features (as features > such as the mocking frameworks will be upstreamed individually), and > so is less useful for the average customer. Until each feature is > upstreamed, it is iterated on rapidly to address comments from the > kernel community, so in-progress features are not stable enough to > reasonably build on. Finally, it exists only as a set of patches on > mailing lists, rather than as a maintained git repository (due to the > fact that the patches themselves are changing rapidly), making it > difficult for early adopters to incorporate into their own trees. > > Whilst we believe there to be enough (at times conflicting) goals > above to justify having multiple versions of KUnit, we want to ensure > that they are meeting their goals, and that we have a process to > ensure that code finds its way into the correct version, that we can > deprecate and remove failed experiments or superseded versions, and > that we can keep pace with upstream kernel releases. > > ============ > The Proposal > ============ > > We propose having two tracks of development: the upstream kernel > (comprising both code that has been upstreamed, and code which is in > the process of being upstreamed -- i.e. is being reviewed on the > mailing lists), and an 'experimental' branch, which contains features > which are yet to be submitted upstream. > > Periodically, the 'experimental' branch will be replaced with a new > 'experimental' branch based on the latest upstream version. This will > allow us to focus on code based on upstream versions by deprecating > earlier, experimental versions of features. The older experimental > branch will stay around, but receive no new features or support: > customers will need to eventually migrate to either the upstream > kernel (if all of the features they require have been upstreamed), or > to the latest 'experimental' branch (if they still require > non-upstreamed features). > > Where practical, 'experimental' branches will try to provide a stable > interface (and/or compatibility layers/shims) such that users can > write tests against an experimental version and expect our changes to > break them infrequently. We don't intend to make any solid guarantee, > however: so users will need to be able to deal with some breakage. > > We expect most changes to KUnit to be submitted directly upstream, and > reserve the 'experimental' branch for features where: > a) the design is yet to be proven by early adopters, and is likely to > change; > b) the feature is large enough that having a solid staging environment > in which several developers can work is necessary; and/or > c) it depends on existing features only available in the 'experimental' branch. > > An example of features suited to the 'experimental' branch are the > mocking systems[2]. > > ===================== > Lifetime of a feature > ===================== > > A new feature (that for one of the above reasons may require some > experimentation before being sent upstream) would typically be > developed as follows: > > 1. An initial version of the feature is developed against the latest > 'experimental' branch. > 2. The feature is tested by users of the 'experimental' branch. > 3. It is revised as needed within the 'experimental' branch. If > breaking changes are required, and supporting some form of > compatibility layer is practical, such a layer is implemented to avoid > breaking existing 'experimental' customers. > 4. Once the feature is deemed ready, it is submitted to the mailing > lists for the upstream kernel. Any compatibility layers for earlier > experimental version are removed -- this is a 'clean' version of the > change. > 5. Development of the change continues on the kernel mailing lists: > any particularly important changes or bugfixes are ported to the > 'experimental' branch if feasible, but complete convergence is not > expected at this stage. > 6. When the feature is accepted upstream, a new 'experimental' branch > is created based on the new upstream kernel. Development on the old > 'experimental' branch ceases. > 7. The upstream version of the feature has now superseded the > experimental version. All future development is done based on the > upstream version, either in the new 'experimental' branch, or directly > against the upstream kernel. > 8. Any customers still using the experimental version of the feature > will need to update their code to work with the upstream version in > order to update to the new (supported) 'experimental' branch, or to > the upstream kernel. > > =========================== > What does this mean for me? > =========================== > > ...as a user? > ------------- > > If you're using the upstream version of KUnit, nothing! Features will > continue to be pushed upstream when they're ready. > > If you're using the 'kunit/alpha/master' branch, then you'll need to > switch to the new 'experimental-<version>' branches. Every so often, > we'll create a new 'experimental-<version>' branch, and you'll need to > rebase your code on top of that to keep receiving updated features and > bugfixes. While we intend to make this as painless as possible, there > may be breaking changes which require you to rework some of your > tests. > > Once a feature exists in the upstream kernel, the version of that > feature in the 'experimental' branches should stabilise somewhat. > > ...as a KUnit developer? > ------------------------ > > For most changes/bugfixes which apply cleanly to the upstream version, > you should directly submit these upstream as before. If it is feasible > and useful to backport them to the latest 'experimental' branch (e.g., > they are bugfixes to code that the experimental branch already has), > please do so. > > For more major features (or anything which depends on > yet-to-be-upstreamed code), you should develop them in the latest > 'experimental' branch. For the most part, you can treat this the same > way as the old 'kunit/alpha/master' branch, but note that changes will > need to be rebased on top of a newer experimental branch periodically. > It may make sense to use this as an opportunity to tidy the code up > somewhat: by removing compatibility code for deprecated features, or > to merge 'fixup' commits. Once the code has been upstreamed, the next > rebase of the 'experimental' branch should only use the upstream > version: there should not be a separate 'experimental' version of > upstream code save for the development of a new 'experimental' feature > or bugfix. > > For more details, see the 'Lifetime of a feature' section above. > > ============= > In Conclusion > ============= > > We'll continue to be working with upstream as before, but we intend to > rename and more explicitly specify the purpose of the > 'kunit/alpha/master' branch as above. This will require a bit more > work from people using and developing against this 'experimental' > branch, but allows us to continue to use it for features such as the > mocking framework while limiting its divergence from upstream. > > Do let us know if this sounds reasonable to you, if it will affect > your use case, or if you have any questions. > > Cheers, > -- David & Brendan > > > [1] > https://kunit.googlesource.com/linux/+/refs/heads/kunit/alpha/master > [2] > https://kunit.googlesource.com/linux/+/kunit/alpha/master/Documentatio > n/test/usage.rst#187 CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient and may contain material that is proprietary, confidential, privileged or otherwise legally protected or restricted under applicable government laws. Any review, disclosure, distributing or other use without expressed permission of the sender is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies without reading, printing, or saving. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-23 21:45 ` Siebert, David @ 2019-09-23 22:26 ` Brendan Higgins 0 siblings, 0 replies; 8+ messages in thread From: Brendan Higgins @ 2019-09-23 22:26 UTC (permalink / raw) To: Siebert, David Cc: David Gow, David Chiang, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, Shuah Khan, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK On Mon, Sep 23, 2019 at 2:45 PM Siebert, David <David.Siebert@l3harris.com> wrote: > > > Right now I am focused on the mocking and getting that to work. I am still have some very strange issues with mocking some function calls including kernel panics. Okay, I just responded to your last email on the topic. > Your plan seems logical but just how useful is it without mocking? It depends on what you are trying to test. We have a number of tests so far that do not require mocking (there are several in the initial upstream patchset itself). A lot more can be tested with the appropriate use of dependency injection. In anycase, we are still planning on upstreaming support for mocking, it is just going to take some time. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-23 21:41 ` Brendan Higgins 2019-09-23 21:45 ` Siebert, David @ 2019-09-23 21:52 ` shuah 2019-09-23 23:00 ` David Gow 1 sibling, 1 reply; 8+ messages in thread From: shuah @ 2019-09-23 21:52 UTC (permalink / raw) To: Brendan Higgins, David Gow Cc: David Chiang, David Siebert, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK, shuah On 9/23/19 3:41 PM, Brendan Higgins wrote: > On Tue, Sep 17, 2019 at 11:41 AM David Gow <davidgow@google.com> wrote: >> >> TL;DR: We expect KUnit to be accepted upstream into Linus' branch in >> the next week or two, and we now need to figure out what we are going >> to do with our non-upstream 'kunit/alpha/master' branch. > > Given that it has been about a week and we haven't heard any comments, > complaints, or concerns about this. I assume that there are no strong > opinions against this, and people will be generally okay with this > strategy. > > As mentioned previously, we are expecting to see KUnit make it into > torvalds/master this merge window (the next week or so), so we will > probably update/rename kunit/alpha/master shortly thereafter. > > Cheers > >> Hello everyone, >> >> We've put together a rough proposal of what we should do with our >> not-upstream branch, known to most people using it as >> 'kunit/alpha/master'[1], now that KUnit's acceptance into mainline >> appears to be imminent (the KUnit MVP patchset is now in linux-next, >> and the merge window just opened). >> >> ========== >> Background >> ========== >> >> KUnit development is currently split between two versions: the >> 'kunit/alpha/master'[1] git branch, and the version being submitted to >> the upstream Linux kernel. While there are some good reasons to >> continue to have two separate versions of KUnit, at present there is >> some uncertainty around the difference between these versions, and in >> which circumstances each version is useful. >> >> At present, the 'kunit/alpha/master' branch serves a few different >> purposes. It is a place for code not-yet-ready for upstream -- such as >> the mocking framework -- while being developed, while also acting as a >> stable version for customers who do not wish to follow along with the >> changes made during the upstreaming process. Adding to the confusion, >> the name 'kunit/alpha/master' refers to an early (alpha) version of >> KUnit, and the version of KUnit being upstreamed has now diverged >> significantly from this version, requiring significant differences in >> documentation, and requiring a number of changes to tests when porting >> from one version to the other. Finally, it is not clear how the >> 'kunit/alpha/master' version should evolve as features it contains are >> upstreamed. >> >> On the other hand, the version being upstreamed has its own >> complications. It contains significantly fewer features (as features >> such as the mocking frameworks will be upstreamed individually), and >> so is less useful for the average customer. Until each feature is >> upstreamed, it is iterated on rapidly to address comments from the >> kernel community, so in-progress features are not stable enough to >> reasonably build on. Finally, it exists only as a set of patches on >> mailing lists, rather than as a maintained git repository (due to the >> fact that the patches themselves are changing rapidly), making it >> difficult for early adopters to incorporate into their own trees. >> >> Whilst we believe there to be enough (at times conflicting) goals >> above to justify having multiple versions of KUnit, we want to ensure >> that they are meeting their goals, and that we have a process to >> ensure that code finds its way into the correct version, that we can >> deprecate and remove failed experiments or superseded versions, and >> that we can keep pace with upstream kernel releases. >> >> ============ >> The Proposal >> ============ >> >> We propose having two tracks of development: the upstream kernel >> (comprising both code that has been upstreamed, and code which is in >> the process of being upstreamed -- i.e. is being reviewed on the >> mailing lists), and an 'experimental' branch, which contains features >> which are yet to be submitted upstream. >> My concern with this approach is either one could outdated. is there a reason continue in parallel mode. I would rathet see development happen upstream so we don't have lot of code to be upstreamed sitting in an experimental branch while upstream keeps moving. It is given that they will diverge. thanks, -- Shuah ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-23 21:52 ` shuah @ 2019-09-23 23:00 ` David Gow 2019-09-23 23:19 ` shuah 0 siblings, 1 reply; 8+ messages in thread From: David Gow @ 2019-09-23 23:00 UTC (permalink / raw) To: shuah Cc: Brendan Higgins, David Chiang, David Siebert, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK On Mon, Sep 23, 2019 at 2:52 PM shuah <shuah@kernel.org> wrote: > > My concern with this approach is either one could outdated. is there a > reason continue in parallel mode. I would rathet see development happen > upstream so we don't have lot of code to be upstreamed sitting in an > experimental branch while upstream keeps moving. It is given that they > will diverge. I definitely appreciate that, and the aim certainly is to have most changes go straight upstream without passing through the 'experimental' branch first. The real purpose of the 'experimental' branch is to have somewhere to keep the mocking functionality before we're ready to upstream it. Given that there are already people using the current version of the mocking framework, we want to provide a smooth-ish path to upstream by providing this branch which is at least closer to upstream than when we are now (and that'll stay as close to upstream as possible through regular rebasing, rather than staying 'stuck' on the older versions). Ultimately, the 'experimental' branch should go away once all of the pieces of the mocking framework have made it upstream, so this is really outlining a transition plan. Now, if we thought that the mocking changes were sufficiently ready, we could try to upstream them soon and kill the old 'kunit/alpha/master' branch without having the intermediate 'experimental' branch. As it is, though, I don't think we feel the mocking framework will be ready for quite some time. So, yes, they'll diverge a bit, but hopefully a bit less than with what we're going now with our 'kunit/master/alpha' branch. Most development that doesn't relate to the mocking system should go directly upstream, and we'll try to minimise the divergence in the 'experimental' branch by rebasing it regularly on top of any upstream changes. When in doubt, though, KUnit changes should definitely be going upstream rather than to this 'experimental' branch. Hopefully that minimises the divergence and makes this more tolerable. Cheers, -- David ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-23 23:00 ` David Gow @ 2019-09-23 23:19 ` shuah 2019-09-24 0:05 ` Brendan Higgins 0 siblings, 1 reply; 8+ messages in thread From: shuah @ 2019-09-23 23:19 UTC (permalink / raw) To: David Gow Cc: Brendan Higgins, David Chiang, David Siebert, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK, shuah On 9/23/19 5:00 PM, David Gow wrote: > On Mon, Sep 23, 2019 at 2:52 PM shuah <shuah@kernel.org> wrote: >> >> My concern with this approach is either one could outdated. is there a >> reason continue in parallel mode. I would rathet see development happen >> upstream so we don't have lot of code to be upstreamed sitting in an >> experimental branch while upstream keeps moving. It is given that they >> will diverge. > > I definitely appreciate that, and the aim certainly is to have most > changes go straight upstream without passing through the > 'experimental' branch first. > > The real purpose of the 'experimental' branch is to have somewhere to > keep the mocking functionality before we're ready to upstream it. > Given that there are already people using the current version of the > mocking framework, we want to provide a smooth-ish path to upstream by > providing this branch which is at least closer to upstream than when > we are now (and that'll stay as close to upstream as possible through > regular rebasing, rather than staying 'stuck' on the older versions). > What I would like to see is a freeze on the experimental branch as soon as KUnit goes into mainline (which is really at the end of this week) Start draining the experimental branch with a goal to get all everything currently staged there mainlined. Please define clear sunset date for the experimental branch. Without that we are looking at a lot of pain in the future. thanks, -- Shuah ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kunit: what do we do with the 'kunit/alpha/master' branch? 2019-09-23 23:19 ` shuah @ 2019-09-24 0:05 ` Brendan Higgins 0 siblings, 0 replies; 8+ messages in thread From: Brendan Higgins @ 2019-09-24 0:05 UTC (permalink / raw) To: shuah Cc: David Gow, David Chiang, David Siebert, Kees Cook, Mike Salvatore, Pei Huang, Sagi Shahar, Sangsu Ha, kunit-dev, open list:KERNEL SELFTEST FRAMEWORK On Mon, Sep 23, 2019 at 05:19:58PM -0600, shuah wrote: > On 9/23/19 5:00 PM, David Gow wrote: > > On Mon, Sep 23, 2019 at 2:52 PM shuah <shuah@kernel.org> wrote: > > > > > > My concern with this approach is either one could outdated. is there a > > > reason continue in parallel mode. I would rathet see development happen > > > upstream so we don't have lot of code to be upstreamed sitting in an > > > experimental branch while upstream keeps moving. It is given that they > > > will diverge. > > > > I definitely appreciate that, and the aim certainly is to have most > > changes go straight upstream without passing through the > > 'experimental' branch first. > > > > The real purpose of the 'experimental' branch is to have somewhere to > > keep the mocking functionality before we're ready to upstream it. > > Given that there are already people using the current version of the > > mocking framework, we want to provide a smooth-ish path to upstream by > > providing this branch which is at least closer to upstream than when > > we are now (and that'll stay as close to upstream as possible through > > regular rebasing, rather than staying 'stuck' on the older versions). > > > > What I would like to see is a freeze on the experimental branch as soon > as KUnit goes into mainline (which is really at the end of this week) > > Start draining the experimental branch with a goal to get all everything > currently staged there mainlined. Yep, that's the plan. Sorry, if that wasn't clear, but we were trying to be intentionally vague about some things to give ourselves room to maneuver. In anycase, we actually want to be pretty aggressive dropping things from experimental as soon as the feature makes it upstream. > Please define clear sunset date for the experimental branch. Without > that we are looking at a lot of pain in the future. I would like to, but I find being able to predict how long it takes to do something upstream to be pretty hard. Especially with large features where people are likely to have strong opinions on. To give you and idea for upstreaming mocking stuff, I see 3 major components: - Basic "class" mocking and parameter matchers (might be able to split them into two parts) - about 7 patches. - Arbitrary function mocking and spying - currently just a couple patches, but I expect a lot of discussion. I am actually hoping we can use some of Knut's work for this. So this is probably a pretty big project. - Platform/hardware mocking and faking - currently also just a handful of patches, but I also expect to get a lot of discussion on this. I could easily see all this taking well over a year; nevertheless, we want to work on other things as well. In fact, from my talk at LPC, it seems like the general consensus is that the mocking stuff is not a very high priority in terms of what the people there wanted to see. So I definitely want to see this all go upstream as soon as possible, nothing would make me happier; however, I think the reality is that there is too much uncertainty to paint a hard deadline. I think it probably makes sense to try to set a roadmap, but I think it will probably end up being pretty rough past Q4. Nevertheless, I am open to alternatives. I know that trying to maintain a staging repo is a lot of work with no long term benefit, and I think any amount of worked saved there can be spent on more useful things. Part of the reason we shared this publicly was that we hoped that smarter more experienced people might be able to save us from some of this pain :-) Looking forward to hearing people's thoughts! ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-09-24 0:05 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-17 18:41 kunit: what do we do with the 'kunit/alpha/master' branch? David Gow 2019-09-23 21:41 ` Brendan Higgins 2019-09-23 21:45 ` Siebert, David 2019-09-23 22:26 ` Brendan Higgins 2019-09-23 21:52 ` shuah 2019-09-23 23:00 ` David Gow 2019-09-23 23:19 ` shuah 2019-09-24 0:05 ` Brendan Higgins
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).