announcing the toaster test framework

announcing the toaster test framework

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

Hi,

I put together a couple of scripts that will monitor the toaster mailing
list, and provide smoke testing on each patchset submitted to review
request.

A very rough form of these scripts existed for some time, but I've taken
time to clean that up, and enough code to be able to add tests very easily,
and make everything a bit more reliable.

It lives outside of the poky/ or bitbake/ trees, and you can check out the
code here, until moved to a proper repo.


http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=yocto-tts/master


Currently, my installation is emailing tests results only to me, but once
I've been satisfied that it doesn't crash or burn the mailing list, I will
update the setup to mail results directly to the mailing list.


You can also use it to manually run tests on your branch prior to
submission.

Cheers,
Alex

-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 1678 bytes --]

Re: announcing the toaster test framework

[Michael Wood]

On 27/04/15 18:22, Damian, Alexandru wrote:
> Hi,
>
> I put together a couple of scripts that will monitor the toaster 
> mailing list, and provide smoke testing on each patchset submitted to 
> review request.
>
> A very rough form of these scripts existed for some time, but I've 
> taken time to clean that up, and enough code to be able to add tests 
> very easily, and make everything a bit more reliable.
>
> It lives outside of the poky/ or bitbake/ trees, and you can check out 
> the code here, until moved to a proper repo.
>
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=yocto-tts/master
>
>
> Currently, my installation is emailing tests results only to me, but 
> once I've been satisfied that it doesn't crash or burn the mailing 
> list, I will update the setup to mail results directly to the mailing 
> list.
>
>
> You can also use it to manually run tests on your branch prior to 
> submission.
>

I know this wasn't an RFC but I do have one question... you're 
encoraging people to write tests for this new framework, but when are we 
saying a test be sent to the "toaster test framework" and when should it 
be a unit test in Toaster?

...My thoughts are that if it's possible to write the unit test in 
toaster with django[1] (i.e no external tools) then that should be the 
primary place for tests, we should be in the habbit of running 
./manage.py test pre-submission of patches. Those tests are broken or 
missing at the moment, so I hope we can fix those first.

What do you think?

[1] e.g. some I wrote for the error-report-web:
http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/tree/Post/test.py



> Cheers,
> Alex
>
> -- 
> Alex Damian
> Yocto Project
> SSG / OTC
>
>

Re: announcing the toaster test framework

[Barros Pena, Belen]

On 28/04/2015 20:54, "Michael Wood" <michael.g.wood@intel.com> wrote:

>On 27/04/15 18:22, Damian, Alexandru wrote:
>> Hi,
>>
>> I put together a couple of scripts that will monitor the toaster
>> mailing list, and provide smoke testing on each patchset submitted to
>> review request.
>>
>> A very rough form of these scripts existed for some time, but I've
>> taken time to clean that up, and enough code to be able to add tests
>> very easily, and make everything a bit more reliable.
>>
>> It lives outside of the poky/ or bitbake/ trees, and you can check out
>> the code here, until moved to a proper repo.
>>
>>
>> 
>>http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=yocto-tts/m
>>aster
>>
>>
>> Currently, my installation is emailing tests results only to me, but
>> once I've been satisfied that it doesn't crash or burn the mailing
>> list, I will update the setup to mail results directly to the mailing
>> list.
>>
>>
>> You can also use it to manually run tests on your branch prior to
>> submission.
>>
>
>I know this wasn't an RFC but I do have one question... you're
>encoraging people to write tests for this new framework, but when are we
>saying a test be sent to the "toaster test framework" and when should it
>be a unit test in Toaster?
>
>...My thoughts are that if it's possible to write the unit test in
>toaster with django[1] (i.e no external tools) then that should be the
>primary place for tests, we should be in the habbit of running
>./manage.py test pre-submission of patches. Those tests are broken or
>missing at the moment, so I hope we can fix those first.

So, we have unit tests in django that developers can run before submitting
their patches. We also have the stuff from Alex, which will run tests on
patches submitted to the mailing list. Both things seem useful, but I
wonder: is there any way we could integrate them into a coherent workflow?
Could we centralise all tests in django, for example, and run them on the
patches landing on the mailing list just in case developers didn't or
didn't catch a failed result?

I am not sure this would make sense, but just in case it does ...

Cheers

Belén


>
>What do you think?
>
>[1] e.g. some I wrote for the error-report-web:
>http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/tree/Post/test.
>py
>
>
>
>> Cheers,
>> Alex
>>
>> -- 
>> Alex Damian
>> Yocto Project
>> SSG / OTC
>>
>>
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: announcing the toaster test framework

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 2491 bytes --]

On Tue, Apr 28, 2015 at 8:54 PM, Michael Wood <michael.g.wood@intel.com>
wrote:

> On 27/04/15 18:22, Damian, Alexandru wrote:
>
>> Hi,
>>
>> I put together a couple of scripts that will monitor the toaster mailing
>> list, and provide smoke testing on each patchset submitted to review
>> request.
>>
>> A very rough form of these scripts existed for some time, but I've taken
>> time to clean that up, and enough code to be able to add tests very easily,
>> and make everything a bit more reliable.
>>
>> It lives outside of the poky/ or bitbake/ trees, and you can check out
>> the code here, until moved to a proper repo.
>>
>>
>>
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=yocto-tts/master
>>
>>
>> Currently, my installation is emailing tests results only to me, but once
>> I've been satisfied that it doesn't crash or burn the mailing list, I will
>> update the setup to mail results directly to the mailing list.
>>
>>
>> You can also use it to manually run tests on your branch prior to
>> submission.
>>
>>
> I know this wasn't an RFC but I do have one question... you're encoraging
> people to write tests for this new framework, but when are we saying a test
> be sent to the "toaster test framework" and when should it be a unit test
> in Toaster?
>
> ...My thoughts are that if it's possible to write the unit test in toaster
> with django[1] (i.e no external tools) then that should be the primary
> place for tests, we should be in the habbit of running ./manage.py test
> pre-submission of patches. Those tests are broken or missing at the moment,
> so I hope we can fix those first.
>
> What do you think?
>

​This would be ideal. I'm preparing a test in the TTS that will run the
Django unit tests, but the TTS is designed to smoke-run (is this a term?)
functional tests before pushing upstream.

It is not intended to replace unit testing, which should occur in Django
tests, as you described. So it's an addition, not a replacement.



>
> [1] e.g. some I wrote for the error-report-web:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/tree/Post/test.py
>
>
>
>
>  Cheers,
>> Alex
>>
>> --
>> Alex Damian
>> Yocto Project
>> SSG / OTC
>>
>>
>>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 4131 bytes --]

Re: announcing the toaster test framework

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 3691 bytes --]

On Wed, Apr 29, 2015 at 10:27 AM, Barros Pena, Belen <
belen.barros.pena@intel.com> wrote:

>
>
> On 28/04/2015 20:54, "Michael Wood" <michael.g.wood@intel.com> wrote:
>
> >On 27/04/15 18:22, Damian, Alexandru wrote:
> >> Hi,
> >>
> >> I put together a couple of scripts that will monitor the toaster
> >> mailing list, and provide smoke testing on each patchset submitted to
> >> review request.
> >>
> >> A very rough form of these scripts existed for some time, but I've
> >> taken time to clean that up, and enough code to be able to add tests
> >> very easily, and make everything a bit more reliable.
> >>
> >> It lives outside of the poky/ or bitbake/ trees, and you can check out
> >> the code here, until moved to a proper repo.
> >>
> >>
> >>
> >>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=yocto-tts/m
> >>aster
> >>
> >>
> >> Currently, my installation is emailing tests results only to me, but
> >> once I've been satisfied that it doesn't crash or burn the mailing
> >> list, I will update the setup to mail results directly to the mailing
> >> list.
> >>
> >>
> >> You can also use it to manually run tests on your branch prior to
> >> submission.
> >>
> >
> >I know this wasn't an RFC but I do have one question... you're
> >encoraging people to write tests for this new framework, but when are we
> >saying a test be sent to the "toaster test framework" and when should it
> >be a unit test in Toaster?
> >
> >...My thoughts are that if it's possible to write the unit test in
> >toaster with django[1] (i.e no external tools) then that should be the
> >primary place for tests, we should be in the habbit of running
> >./manage.py test pre-submission of patches. Those tests are broken or
> >missing at the moment, so I hope we can fix those first.
>
> So, we have unit tests in django that developers can run before submitting
> their patches. We also have the stuff from Alex, which will run tests on
> patches submitted to the mailing list. Both things seem useful, but I
> wonder: is there any way we could integrate them into a coherent workflow?
> Could we centralise all tests in django, for example, and run them on the
> patches landing on the mailing list just in case developers didn't or
> didn't catch a failed result?
>
>
​It's not as bad as you describe, it's a bit worse. There are already two
other places containing Toaster testing: one developed under the
oe-selftest framework, targeted at verifying data collection, etc. and the
UI automated testing using Selenium.

What I'm intending to do is make sure that TTS runs its own tests, Django
unit tests, oe-selftest tests , and the still-pending UI automated tests in
an automated manner just to make sure we have these tests run. It is an
addition to the developer's own testing and to the QA flow, and not a
replacement.



> I am not sure this would make sense, but just in case it does ...
>
> Cheers
>
> Belén
>
>
> >
> >What do you think?
> >
> >[1] e.g. some I wrote for the error-report-web:
> >http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/tree/Post/test
> .
> >py
> >
> >
> >
> >> Cheers,
> >> Alex
> >>
> >> --
> >> Alex Damian
> >> Yocto Project
> >> SSG / OTC
> >>
> >>
> >
> >--
> >_______________________________________________
> >toaster mailing list
> >toaster@yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/toaster
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 5680 bytes --]