KUnit Hackathon

KUnit Hackathon

[André Almeida]

Hello,

We belong to two student groups, FLUSP [1] and LKCAMP [2], both of which
are focused on sharing kernel and free software development knowledge
and experience with fellow free software developers and newcomers.

As part of our efforts, we'll be organizing a KUnit hackathon in the
next Saturday (July 10), where we intend to help newcomers convert
existing runtime tests (the ones found at lib/) to KUnit and maybe
create new ones. Depending on the number of attendees, a high volume of
patches may be sent throughout the day. We will do our best to review
all patches before they go to the kernel mailing lists hoping to avoid
wasting your time with minor patching issues.

So we wanted to let you know of all this beforehand and give you the
time to send any suggestions or comments on all this. For instance, we
may ask people to add a special tag to their patches so you may batch
review them all at a later time if you wish.

Anyhow, we'd really appreciate having your opinion on this.
Thanks!

[1] - https://flusp.ime.usp.br/
[2] - https://lkcamp.dev/

Re: KUnit Hackathon

[Daniel Latypov]

On Mon, Jul 5, 2021 at 1:41 PM André Almeida <andrealmeid@riseup.net> wrote:
>
> Hello,
>
> We belong to two student groups, FLUSP [1] and LKCAMP [2], both of which
> are focused on sharing kernel and free software development knowledge
> and experience with fellow free software developers and newcomers.
>
> As part of our efforts, we'll be organizing a KUnit hackathon in the
> next Saturday (July 10), where we intend to help newcomers convert
> existing runtime tests (the ones found at lib/) to KUnit and maybe
> create new ones. Depending on the number of attendees, a high volume of
> patches may be sent throughout the day. We will do our best to review
> all patches before they go to the kernel mailing lists hoping to avoid
> wasting your time with minor patching issues.

That sounds great!

>
> So we wanted to let you know of all this beforehand and give you the
> time to send any suggestions or comments on all this. For instance, we

Some random, unorganized thoughts:
* I (as a kernel newbie) found the ownership of tests under lib/ to be
a bit unclear.

* Make sure to base on 5.14, there's some new features that could be of use
  * QEMU support in kunit.py: it's no longer as big of a deal if we
can't get tests running under UML. Brendan's made it fairly easy to
run tests in an x86 VM.
  * Skipped tests support: can now mark a test skipped. Some of the
lib/ tests could benefit, e.g. test_hash.c
  * another lib/ test conversion just landed in 5.14, [1], along with
kunit_kcalloc/kunit_malloc_array() helpers
  * Documentation/dev-tools/kunit/running_tips.rst - has instructions
for how to generate code coverage reports using UML/kunit.py
    * This seems like it could be useful... [2]

* I feel that the hackathon will surface a number of feature requests
for KUnit. Some patches might be blocked on these features (like your
uuid patch).
  * Might be good to track these as they come up and note which
patches are blocked.

* Should we make a decision beforehand wrt renaming test files and
kconfig options to match the style guide? In the most recent
conversion [1], I left them as-is. The answer depends on the test
owner, but we should probably be consistent with what we do in the
v1's at least.

* I assume test modules used in selftests (test_bitmap, test_printf,
etc.) should remain untouched for now ?
  * KUnit does *not* return a non-zero exit code when you modprobe the
test, as the userspace part of those tests assumes.
  * We could write a wrapper like kselftest/module.sh, but for KUnit
test modules, but this extra in-flight patch would become a dependency
for maybe multiple other conversions.
  * I'd like it if we had ^ eventually, but I don't know how others
feel about that.

* I had some small patches for converting test_min_heap.c and
test_sort.c that I should get around to sending out.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ebd09577be6c15ee2d343cf60e5bb819946a5ee8
[2] I currently see
  Overall coverage rate:
  lines......: 16.4% (20034 of 122481 lines)
  functions..: 18.1% (2032 of 11205 functions)
with the following kunitconfig
  CONFIG_KUNIT=y
  CONFIG_KUNIT_ALL_TESTS=y
  CONFIG_DEBUG_KERNEL=y
  CONFIG_DEBUG_INFO=y
  CONFIG_GCOV=y

This might be a neat way to motivate participants by showing #s going up.
Note: I'm more interested in the # of covered functions than anything else.
The %s are nice to look at, but not that useful since we're only
compiling a small fraction of kernel code.

> may ask people to add a special tag to their patches so you may batch
> review them all at a later time if you wish.
>
> Anyhow, we'd really appreciate having your opinion on this.
> Thanks!
>
> [1] - https://flusp.ime.usp.br/
> [2] - https://lkcamp.dev/
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/83ff3ed3-e004-29b2-b23c-a2ba4d5a3f7f%40riseup.net.

Re: KUnit Hackathon

[André Almeida]

Hi Daniel,

Às 03:48 de 07/07/21, Daniel Latypov escreveu:
> On Mon, Jul 5, 2021 at 1:41 PM André Almeida <andrealmeid@riseup.net> wrote:
>>
>> Hello,
>>
>> We belong to two student groups, FLUSP [1] and LKCAMP [2], both of which
>> are focused on sharing kernel and free software development knowledge
>> and experience with fellow free software developers and newcomers.
>>
>> As part of our efforts, we'll be organizing a KUnit hackathon in the
>> next Saturday (July 10), where we intend to help newcomers convert
>> existing runtime tests (the ones found at lib/) to KUnit and maybe
>> create new ones. Depending on the number of attendees, a high volume of
>> patches may be sent throughout the day. We will do our best to review
>> all patches before they go to the kernel mailing lists hoping to avoid
>> wasting your time with minor patching issues.
> 
> That sounds great!
> 

:)

>>
>> So we wanted to let you know of all this beforehand and give you the
>> time to send any suggestions or comments on all this. For instance, we
> 
> Some random, unorganized thoughts:
> * I (as a kernel newbie) found the ownership of tests under lib/ to be
> a bit unclear.
> 

Indeed, but we found that tests under lib/ are the easiest ones to
convert, given that they deal with more "generic" programming aspects,
like linked lists and sort, and doesn't requires subsystem specific
knowledge.

> * Make sure to base on 5.14, there's some new features that could be of use
>   * QEMU support in kunit.py: it's no longer as big of a deal if we
> can't get tests running under UML. Brendan's made it fairly easy to
> run tests in an x86 VM.
>   * Skipped tests support: can now mark a test skipped. Some of the
> lib/ tests could benefit, e.g. test_hash.c
>   * another lib/ test conversion just landed in 5.14, [1], along with
> kunit_kcalloc/kunit_malloc_array() helpers
>   * Documentation/dev-tools/kunit/running_tips.rst - has instructions
> for how to generate code coverage reports using UML/kunit.py
>     * This seems like it could be useful... [2]
> 

Thanks for those info! I wasn't aware of those features.

> * I feel that the hackathon will surface a number of feature requests
> for KUnit. Some patches might be blocked on these features (like your
> uuid patch).
>   * Might be good to track these as they come up and note which
> patches are blocked.
> 

Good suggestion, I'll try to track this.

> * Should we make a decision beforehand wrt renaming test files and
> kconfig options to match the style guide? In the most recent
> conversion [1], I left them as-is. The answer depends on the test
> owner, but we should probably be consistent with what we do in the
> v1's at least.
> 

I believe that to left as-is is easier, and we will suggest people to do
that.

> * I assume test modules used in selftests (test_bitmap, test_printf,
> etc.) should remain untouched for now ?
>   * KUnit does *not* return a non-zero exit code when you modprobe the
> test, as the userspace part of those tests assumes.
>   * We could write a wrapper like kselftest/module.sh, but for KUnit
> test modules, but this extra in-flight patch would become a dependency
> for maybe multiple other conversions.
>   * I'd like it if we had ^ eventually, but I don't know how others
> feel about that.
> 

Yeah, I don't think we would be able to deal with all of this in this
hackathon, so we won't touch those tests for now.

> * I had some small patches for converting test_min_heap.c and
> test_sort.c that I should get around to sending out.
> 

So this means that we shouldn't convert them, right?

> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ebd09577be6c15ee2d343cf60e5bb819946a5ee8
> [2] I currently see
>   Overall coverage rate:
>   lines......: 16.4% (20034 of 122481 lines)
>   functions..: 18.1% (2032 of 11205 functions)
> with the following kunitconfig
>   CONFIG_KUNIT=y
>   CONFIG_KUNIT_ALL_TESTS=y
>   CONFIG_DEBUG_KERNEL=y
>   CONFIG_DEBUG_INFO=y
>   CONFIG_GCOV=y
> 
> This might be a neat way to motivate participants by showing #s going up.
> Note: I'm more interested in the # of covered functions than anything else.
> The %s are nice to look at, but not that useful since we're only
> compiling a small fraction of kernel code.

Cool! In the end of the hackathon we will compare the numbers to see how
much we were able to do in the event.

> 
>> may ask people to add a special tag to their patches so you may batch
>> review them all at a later time if you wish.
>>
>> Anyhow, we'd really appreciate having your opinion on this.
>> Thanks!
>>
>> [1] - https://flusp.ime.usp.br/
>> [2] - https://lkcamp.dev/
>>
>> --
>> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/83ff3ed3-e004-29b2-b23c-a2ba4d5a3f7f%40riseup.net.

Re: KUnit Hackathon

[Daniel Latypov]

On Wed, Jul 7, 2021 at 5:23 PM André Almeida <andrealmeid@riseup.net> wrote:
>
> Hi Daniel,
>
> Às 03:48 de 07/07/21, Daniel Latypov escreveu:
> > On Mon, Jul 5, 2021 at 1:41 PM André Almeida <andrealmeid@riseup.net> wrote:
> >>
> >> Hello,
> >>
> >> We belong to two student groups, FLUSP [1] and LKCAMP [2], both of which
> >> are focused on sharing kernel and free software development knowledge
> >> and experience with fellow free software developers and newcomers.
> >>
> >> As part of our efforts, we'll be organizing a KUnit hackathon in the
> >> next Saturday (July 10), where we intend to help newcomers convert
> >> existing runtime tests (the ones found at lib/) to KUnit and maybe
> >> create new ones. Depending on the number of attendees, a high volume of
> >> patches may be sent throughout the day. We will do our best to review
> >> all patches before they go to the kernel mailing lists hoping to avoid
> >> wasting your time with minor patching issues.
> >
> > That sounds great!
> >
>
> :)
>
> >>
> >> So we wanted to let you know of all this beforehand and give you the
> >> time to send any suggestions or comments on all this. For instance, we
> >
> > Some random, unorganized thoughts:
> > * I (as a kernel newbie) found the ownership of tests under lib/ to be
> > a bit unclear.
> >
>
> Indeed, but we found that tests under lib/ are the easiest ones to
> convert, given that they deal with more "generic" programming aspects,
> like linked lists and sort, and doesn't requires subsystem specific
> knowledge.

Yeah, I think these are the best for a hackathon.

Plus they're a better fit for KUnit right now, since they're the more
standalone bits of code.
Writing tests for more subsystem-specific code can require a lot of
groundwork, e.g. davidgow@google.com is trudging through what needs to
be set in a superblock to write ext4 tests.
Eventually, we'd ideally have some chunks of shared code to setup
subsystem-specific structures for tests, but they don't exist yet.

I didn't articulate it well, but my concern was finding the "right"
people to review the patches.
It feels like an unfortunate risk that we'll have to take, having some
patches potentially sit unreviewed for a while.

>
> > * Make sure to base on 5.14, there's some new features that could be of use
> >   * QEMU support in kunit.py: it's no longer as big of a deal if we
> > can't get tests running under UML. Brendan's made it fairly easy to
> > run tests in an x86 VM.
> >   * Skipped tests support: can now mark a test skipped. Some of the
> > lib/ tests could benefit, e.g. test_hash.c
> >   * another lib/ test conversion just landed in 5.14, [1], along with
> > kunit_kcalloc/kunit_malloc_array() helpers
> >   * Documentation/dev-tools/kunit/running_tips.rst - has instructions
> > for how to generate code coverage reports using UML/kunit.py
> >     * This seems like it could be useful... [2]
> >
>
> Thanks for those info! I wasn't aware of those features.
>
> > * I feel that the hackathon will surface a number of feature requests
> > for KUnit. Some patches might be blocked on these features (like your
> > uuid patch).
> >   * Might be good to track these as they come up and note which
> > patches are blocked.
> >
>
> Good suggestion, I'll try to track this.

Thanks!
Hopefully we can try and help get some of these resolved, but it might
take a while.

>
> > * Should we make a decision beforehand wrt renaming test files and
> > kconfig options to match the style guide? In the most recent
> > conversion [1], I left them as-is. The answer depends on the test
> > owner, but we should probably be consistent with what we do in the
> > v1's at least.
> >
>
> I believe that to left as-is is easier, and we will suggest people to do
> that.
>
> > * I assume test modules used in selftests (test_bitmap, test_printf,
> > etc.) should remain untouched for now ?
> >   * KUnit does *not* return a non-zero exit code when you modprobe the
> > test, as the userspace part of those tests assumes.
> >   * We could write a wrapper like kselftest/module.sh, but for KUnit
> > test modules, but this extra in-flight patch would become a dependency
> > for maybe multiple other conversions.
> >   * I'd like it if we had ^ eventually, but I don't know how others
> > feel about that.
> >
>
> Yeah, I don't think we would be able to deal with all of this in this
> hackathon, so we won't touch those tests for now.
>
> > * I had some small patches for converting test_min_heap.c and
> > test_sort.c that I should get around to sending out.
> >
>
> So this means that we shouldn't convert them, right?

Yeah, you can leave those ones to me.
I'll try and send them out real soon so people can just check the
mailing list to see what's in flight.

I'm not aware of anyone else working on converting the other tests
right now, so everything else should be fair game.

>
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ebd09577be6c15ee2d343cf60e5bb819946a5ee8
> > [2] I currently see
> >   Overall coverage rate:
> >   lines......: 16.4% (20034 of 122481 lines)
> >   functions..: 18.1% (2032 of 11205 functions)
> > with the following kunitconfig
> >   CONFIG_KUNIT=y
> >   CONFIG_KUNIT_ALL_TESTS=y
> >   CONFIG_DEBUG_KERNEL=y
> >   CONFIG_DEBUG_INFO=y
> >   CONFIG_GCOV=y
> >
> > This might be a neat way to motivate participants by showing #s going up.
> > Note: I'm more interested in the # of covered functions than anything else.
> > The %s are nice to look at, but not that useful since we're only
> > compiling a small fraction of kernel code.
>
> Cool! In the end of the hackathon we will compare the numbers to see how
> much we were able to do in the event.
>
> >
> >> may ask people to add a special tag to their patches so you may batch
> >> review them all at a later time if you wish.
> >>
> >> Anyhow, we'd really appreciate having your opinion on this.
> >> Thanks!
> >>
> >> [1] - https://flusp.ime.usp.br/
> >> [2] - https://lkcamp.dev/
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> >> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/83ff3ed3-e004-29b2-b23c-a2ba4d5a3f7f%40riseup.net.

Re: KUnit Hackathon

[Brendan Higgins]

On Mon, Jul 5, 2021 at 1:41 PM André Almeida <andrealmeid@riseup.net> wrote:
>
> Hello,
>
> We belong to two student groups, FLUSP [1] and LKCAMP [2], both of which
> are focused on sharing kernel and free software development knowledge
> and experience with fellow free software developers and newcomers.
>
> As part of our efforts, we'll be organizing a KUnit hackathon in the
> next Saturday (July 10), where we intend to help newcomers convert
> existing runtime tests (the ones found at lib/) to KUnit and maybe
> create new ones. Depending on the number of attendees, a high volume of
> patches may be sent throughout the day. We will do our best to review
> all patches before they go to the kernel mailing lists hoping to avoid
> wasting your time with minor patching issues.

That's awesome! I'm really excited!

> So we wanted to let you know of all this beforehand and give you the
> time to send any suggestions or comments on all this. For instance, we
> may ask people to add a special tag to their patches so you may batch
> review them all at a later time if you wish.

I wouldn't worry about it until it becomes an actual "problem"; too
many patches is a good problem to have :-)

> Anyhow, we'd really appreciate having your opinion on this.

I don't think I have anything to add that Daniel didn't already add in
his reply. Also, I have seen Marcelo around for a while. I have to
say, I don't think I have seen a student group this involved upstream.
I am impressed :-)

> Thanks!

Thank *you*!

I wish I could offer more material assistance, but with only a week's
notice, I don't think I can. I will try to be online periodically
throughout the day, but I am afraid I cannot promise anything.

No matter what, we would be somewhat limited with what we can do in
2021 because of COVID. No promises, but if this goes well maybe next
year we might be able to send someone in person if that's something
you would be interested in.

Best of luck! I hope it goes well!

> [1] - https://flusp.ime.usp.br/
> [2] - https://lkcamp.dev/

Re: KUnit Hackathon

[André Almeida]

Hi Brendan,

Às 17:23 de 07/07/21, Brendan Higgins escreveu:
> On Mon, Jul 5, 2021 at 1:41 PM André Almeida <andrealmeid@riseup.net> wrote:
>>
>> Hello,
>>
>> We belong to two student groups, FLUSP [1] and LKCAMP [2], both of which
>> are focused on sharing kernel and free software development knowledge
>> and experience with fellow free software developers and newcomers.
>>
>> As part of our efforts, we'll be organizing a KUnit hackathon in the
>> next Saturday (July 10), where we intend to help newcomers convert
>> existing runtime tests (the ones found at lib/) to KUnit and maybe
>> create new ones. Depending on the number of attendees, a high volume of
>> patches may be sent throughout the day. We will do our best to review
>> all patches before they go to the kernel mailing lists hoping to avoid
>> wasting your time with minor patching issues.
> 
> That's awesome! I'm really excited!

:D

> 
>> So we wanted to let you know of all this beforehand and give you the
>> time to send any suggestions or comments on all this. For instance, we
>> may ask people to add a special tag to their patches so you may batch
>> review them all at a later time if you wish.
> 
> I wouldn't worry about it until it becomes an actual "problem"; too
> many patches is a good problem to have :-)
> 
>> Anyhow, we'd really appreciate having your opinion on this.
> 
> I don't think I have anything to add that Daniel didn't already add in
> his reply. Also, I have seen Marcelo around for a while. I have to
> say, I don't think I have seen a student group this involved upstream.
> I am impressed :-)
> 
>> Thanks!
> 
> Thank *you*!
> 
> I wish I could offer more material assistance, but with only a week's
> notice, I don't think I can. I will try to be online periodically
> throughout the day, but I am afraid I cannot promise anything.
> 
> No matter what, we would be somewhat limited with what we can do in
> 2021 because of COVID. No promises, but if this goes well maybe next
> year we might be able to send someone in person if that's something
> you would be interested in.

Wow, that would be really awesome!

> 
> Best of luck! I hope it goes well!

Thank you for the support! Next time, I'll make sure to send this notice
earlier

> 
>> [1] - https://flusp.ime.usp.br/
>> [2] - https://lkcamp.dev/